Systems and methods relating to document and fastener identification

ABSTRACT

Method and systems of automated document processing described herein include activating in sequence a plurality of illumination modules of an illumination source to illuminate a document, where the plurality of illumination modules are located at different positions relative to the document. The document can be imaged each time the document is illuminated by an illumination module to provide a plurality of images. A shadow profile of the document can be obtained based on the plurality of images. One or more of a boundary of the document and presence of a fastener attached to the document can be identified using the shadow profile. Any fasteners present may be removed using a robot arm.

CROSS-REFERENCE

This application is a continuation of U.S. patent application Ser. No.16/944,364, filed Jul. 31, 2020, which is a continuation of U.S. patentapplication Ser. No. 16/708,700, filed Dec. 10, 2019, which is acontinuation of U.S. patent application Ser. No. 16/007,069, filed Jun.13, 2018, which is a continuation of International Patent ApplicationNo. PCT/US2016/67583, filed Dec. 19, 2016, which claims the benefit ofU.S. Provisional Application No. 62/233,934, filed Dec. 19, 2015,entitled “Method and Apparatus Utilizing Computer Vision System toIdentify Document Fasteners”, U.S. Provisional Application No.62/233,938, filed Dec. 19, 2015, entitled “Method and ApparatusUtilizing Robotic Automation to Remove Document Fasteners”, and U.S.Provisional Application No. 62/233,947, filed Dec. 19, 2015, entitled“Multi-function Robotic Apparatus for Document Preparation and Imaging”,each of which is hereby incorporated herein by reference in itsentirety.

BACKGROUND

Information can often be stored on physical document files. Suchphysical document files may be packed away in archives, warehousesand/or vaults. Access of information stored on physical document filesmay be costly and time consuming. Physical documents may bedisorganized. Often the documents may include one or more fastenersattached thereto. Not only can the physical documents be difficult tolocate and transport, but once the physical documents have beenidentified, extracting the information from the physical files can alsobe time consuming as well.

SUMMARY

According to some aspects, a method of document processing can compriseactivating, in sequence, a plurality of illumination modules of anillumination source to illuminate a document, wherein the plurality ofillumination modules are located at different positions relative to thedocument; imaging the document each time the document is illuminated byan illumination module of the plurality of illumination modules, toprovide a plurality of images; obtaining a shadow profile of thedocument based on the plurality of images; and identifying, with aid ofat least one processor, at least one of a boundary of the document andpresence of a fastener attached to the document using the shadowprofile.

In some embodiments, identifying the boundary of the document comprisesidentifying an edge having a thickness of less than 0.1 mm. In someembodiments, identifying the presence of the fastener comprisesrecognizing when the fastener is an inverted fastener.

In some embodiments, the method includes, when the fastener is theinverted fastener: applying a suction force upon a portion of thedocument using a suction applicator to lift the document from a documentpedestal; positioning a first document platform under the lifteddocument; positioning a second document platform over the lifteddocument to place the document between and in contact with the firstdocument platform and the second document platform; and rotating thefirst document platform and the second document platform around ahorizontal axis to flip the document over.

In some embodiments, removing the identified fastener comprises:coupling a document transfer end effector to the robot arm; contacting atop surface of the document with the document transfer end effectorcoupled to the robot arm to apply a suction force upon the document tolift the document from a document pedestal on which the document ispositioned to separate the document from other documents on the documentpedestal; rotating a document pincher of the document transfer endeffector to contact a bottom surface of the document to secure thedocument against the document transfer end effector; positioning adocument platform under the lifted document to maintain separation ofthe document from the other documents; releasing the suction forceapplied upon the document by the document transfer end effector to placethe document on the document platform; and releasing the documenttransfer end effector from the robot arm and coupling a fastener removalend effector to the robot arm.

In some embodiments, the method includes: contacting a top surface ofthe document with a document contacting portion of the fastener removalend effector to hold the document against the document platform;inserting a fastener removal tab of the fastener removal end effectorbetween the fastener and the top surface of the document; gripping thefastener with a fastener gripper of the fastener removal end effector toposition at least a portion of the fastener between the fastener inserttab and the fastener gripper; and moving the fastener gripper and thefastener removal tab away from the document to remove the fastener.

In some embodiments, the document further comprises a staple attachedthereto, and wherein identifying the at least one of the boundary of thedocument and presence of a fastener comprises identifying presence ofthe staple.

In some embodiments, the method includes identifying a correspondingfastener removal end effector for a robot arm based on type informationof the fastener attached to the document.

According to some aspects, a document processing system can comprise: adocument pedestal configured to receive a document; an illuminatingsource comprising a plurality of illumination modules configured toilluminate the document, wherein each of the plurality of illuminationmodules are located at different positions relative to the documentpedestal and configured to be illuminated in sequence to illuminate thedocument; an imaging device configured to capture a plurality of images,the plurality of images comprising at least one image of the documenteach time the document is illuminated by each of the plurality ofillumination modules; and a controller unit configured to obtain ashadow profile of the document based on the plurality of images, andidentify at least one of a boundary of the document and presence of afastener attached to the document using the shadow profile.

In some embodiments, the plurality of illumination modules each comprisea plurality of LEDs.

In some embodiments, the system includes a document transferend-effector comprising: a body; a suction applicator extending from alower surface of the body, wherein the suction applicator is configuredto apply a suction force upon at least a portion of a document to liftthe document; and a document pincher comprising a rotatable armconfigured to be rotated towards the body to contact the document andpinch the document between the document pincher and a lower surface ofthe body.

In some embodiments, the system includes a document platform configuredto be positioned beneath the lifted document to separate the documentfrom other documents on the pedestal. In some embodiments, thecontroller unit is configured to identify presence of an invertedfastener, the system further comprising: a first document platformconfigured to be positioned beneath the lifted document to separate thedocument from other documents on the pedestal; and a second documentplatform configured to be positioned over the lifted document, whereinthe document is sandwiched between the first and second documentplatforms, and wherein the first and second document platforms areconfigured to be rotated around a horizontal axis to flip the documentover.

In some embodiments, the system comprises a robot arm and wherein thecontroller unit is configured to instruct removal of the fastener usingan end effector coupled to the robot arm.

In some embodiments, the system comprises a fastener removal endeffector, the fastener removal end effector comprising: a pair ofdocument contacting portion configured to contact a top surface of thedocument to hold the document in place; a fastener removal tab betweenthe pair of document contacting portions, wherein the fastener removaltab is configured to be inserted between the fastener and the topsurface of the document; and a fastener gripper above the fastenerremoval tab configured to grip the fastener to position at least aportion of the fastener between the fastener insert tab and the fastenergripper, wherein movement of the fastener gripper and the fastenerinsert tab away from the document removes the fastener.

In some embodiments, the fastener is a staple and wherein the at leastone of the removal tab and the gripper comprises a width that is thesame as a corresponding width of a crown of the staple.

According to some aspects, an automated system for document processingcomprises: a document input configured to receive a document, whereinthe document comprises a fastener attached thereto; a detection unitconfigured to collect information regarding the document and thefastener; a controller unit configured to receive the information fromthe detection unit to generate a processed information; and aninteraction unit configured to manipulate at least one of the documentand the fastener using a tool selected based on the processedinformation.

In some embodiments, the detection unit comprises: an illuminationsource comprising a plurality of illumination modules configured toilluminate the document and the fastener in sequence from differentpositions; and an image capture device configured to capture a pluralityof images of the document and the fastener, wherein the plurality ofimages comprise at least one image of the document and the fastener eachtime an illumination module of the plurality of illumination modulesilluminates the document and the fastener, wherein the controller isconfigured to compare the plurality of images to generate the processedinformation.

In some embodiments, the tool comprises a fastener removal tool, whereinthe interaction unit is configured to be releasably coupled to the tool,and wherein the controller is configured to instruct the interactionunit to couple to the tool for manipulating at least one of the documentand the fastener to remove the fastener.

In some embodiments, the controller is configured to use the processedinformation to determine a characteristic of at least one of thedocument and the fastener, and wherein the controller is configured toautonomously control, based on the characteristics, a movement of theinteraction unit for manipulating at least one of the document and thefastener. wherein the first portion is fixed along the first portion ofthe motor in a direction substantially parallel to an axis of rotationof the motor.

Additional aspects and advantages of the present disclosure will becomereadily apparent to those skilled in this art from the followingdetailed description, wherein only exemplary embodiments of the presentdisclosure are shown and described, simply by way of illustration of thebest mode contemplated for carrying out the present disclosure. As willbe realized, the present disclosure is capable of other and differentembodiments, and its several details are capable of modifications invarious obvious respects, all without departing from the disclosure.Accordingly, the drawings and description are to be regarded asillustrative in nature, and not as restrictive.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the features and advantages of the presentinvention will be obtained by reference to the following detaileddescription that sets forth illustrative embodiments, in which theprinciples of the invention are utilized, and the accompanying drawingsof which:

FIG. 1 is a schematic diagram of an example of an identification system.

FIG. 2 is a schematic diagram of an example of a document and fasteneridentification system.

FIGS. 3A through 3E are preprocessed and processed images generated bythe document and fastener identification system of FIG. 2.

FIG. 4 is a schematic diagram of an example of document transfer endeffector.

FIGS. 5A and 5B are schematic diagrams of the front view and side view,respectively, of an example of a fastener removal end effector.

FIG. 6 is a schematic diagram of another example of an end effector.

FIG. 7 is a schematic diagram of an example of an infrared fasteneridentification system.

FIG. 8 is a schematic diagram of an example of a lift and scan fasteneridentification system.

FIG. 9 is a schematic diagram of an example of an inductive sensingfastener identification system.

FIG. 10 is a process flow diagram of an example process for identifyinga single sheet of document and/or a fastener.

FIG. 11 is a process flow diagram of an example of a process forremoving a fastener.

FIG. 12 is a process flow diagram of another example of a process forremoving a fastener.

DETAILED DESCRIPTION

While some embodiments of the invention have been shown and describedherein, it will be obvious to those skilled in the art that suchembodiments are provided by way of example only. Numerous variations,changes, and substitutions will now occur to those skilled in the artwithout departing from the invention. It should be understood thatvarious alternatives to the embodiments of the invention describedherein may be employed in practicing the invention.

Immense amounts of information are often stored on physical documents,both by governmental agencies and private entities. Such documents canbe kept in massive archives, warehouses, and/or vaults. The documentsmay or may not have been presorted. A stack of the documents can includedocuments of various sizes, shapes, thicknesses, materials, and/or mayinclude documents oriented differently relative to other documents inthe stack. The documents may or may not have fasteners attached thereto.A wide range of different types of fasteners may be used. Documents canbe packaged in various manners, and/or stored within different types ofcontainers. As an example, a Bankers Box® having a size of about 10inches by 12 inches by 15 inches can store up to about 2,500 pieces ofpaper, and weigh about 25 pounds or more. An estimated of more than 6billion of these types of boxes can be in storage, containing documentswith information stored thereon that can be immensely difficult toaccess.

Physical document files may be difficult to locate, search, and/ormanipulate. Reliance upon manual methods to locate, access, manipulateand/or extract the information from the documents can be costly,unreliable and/or subject to privacy concerns. For example, humanhandling of large amounts of documents may be slow, prone to error,damaging to human health due to exposure of chemical in the documentsand/or fasteners, and/or raise security concerns when documentscontaining sensitive information are involved. Locating particulardocuments and/or transportation of such documents may be laborious andcostly. Storage of information on physical documents may also be proneto loss, such as when the physical documents are lost, destroyed and/ormislabeled.

For at least one or more of the reasons described herein, extraction ofinformation stored on physical documents may be difficult, therebyhindering or preventing access to an immense amount of storedinformation.

Systems and methods described herein relate to automating one or moreof: (1) identifying a single sheet of physical document, such as from astack of physical documents, (2) identifying the type of any fastenersattached to one or more sheets of physical documents, and (3) removal ofany fasteners using appropriate removal tools based on the fastenertype. Such systems and methods may be used to automate digitization ofinformation stored on physical documents. The systems and methods may beincorporated into an automated end-to-end process which includes, forexample, removal of documents from containers, manipulating thedocuments such that information stored on the documents can beextracted, extracting the information from the documents, andrepackaging the documents for further processing. The automatedend-to-end process may include more or fewer steps and the steps can beperformed in various sequences.

Systems and methods described herein relating to automatingidentification of a single sheet of document, any fasteners attachedthereto, and/or removal of the fasteners, can facilitate manipulation ofthe single sheet of document such that information stored on thedocument can be digitally scanned and digitally stored. Conversion ofthe information stored on physical documents to digital information canmake the information easily accessible through digital searching. Thedigitized information can be stored on a remote server at a securedlocation, and/or may be easily and reliably duplicated upon request.Digital information may be more easily searched than information storedon physical documents, and can be easily sent to a target destination.Access of digital information may not involve any human intermediaries,thereby reducing or eliminating accidental or undesired access ofsensitive information to individuals without access authority.Automation of digitization of the information can improve reliability,reduce costs, and/or avoid exposure of sensitive information to humanerrors. An automated process can facilitate increased speed andreliability in handling large volumes of documents.

Correctly detecting the presence of fasteners and/or identifying thetype of fasteners, followed by successful removal of the fasteners, canbe a complex task. Manual identification and removal of the fastenerscan be complicated, tedious and time-consuming. Removal of differenttypes of fasteners can involve use of numerous types of fastener removaltools and avoiding damage to documents may be difficult. Automatedidentification of fasteners may involve accurately determining whetherfeatures are false positives, such as accurately identifying photocopiedfasteners which appear on the documents being processed.

A document and fastener identification system as described herein can beconfigured to identify a single sheet of document, such as from a stackof documents, identify presence of any document fastener devices on adocument or stack of documents, identify the type of the fastenerpresent, and/or removal of the fasteners using the appropriate tool. Thesystem can be configured for automated identification of the singlesheet of document, presence and type of any document fasteners, and/orremoval of the fasteners. The system can be configured to manipulate awide variety of document types containing written and/or printedinformation to prepare the documents for imaging such that theinformation stored on the documents can be stored in digitalrepositories. For example, the separated sheets of documents can besubsequently scanned such that information printed and/or writtenthereon can be digitally captured. In some embodiments, one or moresystems described herein can be configured to separate grouped,attached, stacked and/or bound records into individual sheets. The typesof documents which can be processed by one or more systems describedherein can include, but are not limited to, legal, financial, and/orhistoric records. One or more of the systems described herein can beconfigured to process documents made of various materials, havingvarious sizes, shapes and/or thicknesses.

Systems described herein can be used to remove a variety of fasteners.Such a system can automate identification and removal of fasteners,providing a cost effective, time efficient and reliable method ofpreparing documents such that information stored thereon can bedigitized.

In some embodiments, one or more systems described herein can be suitedfor removal of staples. Staples may be particularly challenging toidentify and/or remove due to its relatively small size and delicatestructure. Due to its small size, a staple present on a document may bemissed and/or errors in detection systems may falsely indicate presenceof a staple when none is actually present. Removal of staples may bedifficult due to the tendency of one or more of its legs of to breakwhen the staple is pulled out from the document. One or more systemsdescribed herein can be configured to provide accurate identificationand reliable removal of staples.

Referring to FIG. 1, a schematic diagram of an example of anidentification system 100 is shown. A target material may be provided tothe identification system for processing. The target material maycomprise one or more documents. A single document may be presented.

Alternatively or in addition, multiple documents may be presented as astack, pile, or any other manner. The identification system can compriseone or more components to collect information regarding the targetmaterial. The identification system can use this information to identifya sheet of document in the target material, and/or presence and/orcharacteristics of any fasteners on the target material. In someembodiments, the identification system can include one or morecomponents to interact with the target material, including any fastenersattached thereto, so as to achieve removal of the fasteners.

The identification system 100 can be configured to identify one or moresheets of a document, and/or information pertaining to any fasteners ona document. The document may be a single sheet or may comprise multiplesheets that may be fastened together with aid of one or more fasteners.The sheet may be a sheet of paper or any other media type, as providedin detail elsewhere herein. The document may be identified and/ordifferentiated from other documents. A single sheet of a document may beidentified and/or differentiated from other sheets within the document,or other sheets of other documents. A boundary (e.g., edge) of a sheetmay be identified. The boundary of the sheet may be useful for detectingand differentiating the sheet from other sheets.

A document may comprise one or more fasteners. Alternatively, thedocument may not have a fastener. The identification system 100 may beconfigured to collect information pertaining to any fasteners on adocument, such as the presence or absence of any fasteners on thedocument, type of fastener, location of fastener, dimensions offastener, or any other information pertaining to the fastener.

The identification system 100 can include a detection unit 102, acontroller unit 104, and/or an interaction unit 106. The detection unitcan be configured to collect information regarding the target materialto identify a single sheet of document, and/or collect informationpertaining to the fasteners. For example, the detection unit can includeone or more detector components configured to collect informationrelating to the target material and/or fasteners attached to the targetmaterial. The detection unit and the controller unit can be inelectrical communication with one another such that the detection unitcan provide the collected information to the controller unit foranalysis. The detection unit can receive instructions from thecontroller unit based on information collected by the detection unit.The controller unit may also be in electrical communication withinteraction unit such that the controller unit can instruct theinteraction unit to perform one or more tasks based on the informationgathered by the detection unit. The interaction unit can be configuredto receive instructions from the controller unit and perform one or moretasks to enable identification of the single sheet of document and/orone or more fasteners attached to the target material, and/or removal ofthe fasteners.

Although the detection unit 102 and the interaction unit 106 aredescribed herein as being a part of one system, it will be understoodthat the detection unit and the interaction unit can be a part ofseparate system. For example, collecting information regarding adocument and/or any fasteners thereon can be automated using a systemdifferent from an automated system configured to interact with thedocument and/or fasteners.

A target material can be presented to the identification system 100. Theidentification system can be configured to receive the target materialpresented in various forms. The target material may comprise onedocument or a stack of documents. The document or stack of documents maybe presented to the identification system manually by an operator of thesystem and/or via an automated process. The document or stack ofdocuments may be presented as a single sheet of document, a plurality ofsheets of documents, a single stack of documents, a plurality of stacksof documents. The document or document stack may be presented in acontainer, such as a tray, a box (e.g., Bankers Boxes®), a bin, and/or afolder. For example, a container containing a document stack can beloaded onto a receiving port of the identification system and thecontainer can be moved by the system to a desired height to transfer oneor more of the documents from within the container to for processing.The system may comprise one or more mechanical components to move thecontainer vertically and/or laterally, such as via tracks over which thecontainer is placed. A desired height of the container can be presetand/or determined using one or more sensors known to one skilled in theart. In some embodiments, the document or document stack may bepresented without any type of container. A document stack presented tothe identification system may not be pre-sorted. For example, thedocument stack can be vertically stacked without any particular sequencein the stacking. In some embodiments, the document stack may bepre-sorted.

Any description herein of a stack of documents may apply to a pluralityof documents presented in any manner. In some embodiments, the stack ofdocuments can comprise documents having a wide variety ofcharacteristics. The stack of documents may comprise documents ofdifferent materials shapes, sizes, and/or thicknesses. The documents maybe of different types from one another. In some instances, varioussheets within a document may be of different types from one another.Different types of documents may have different characteristics. Thestack of documents may comprise one or more documents which have anadhesive on at least a portion thereof. In some embodiments, one or moredocuments of the stack processed by the identification system 100 may beoriented differently from one or more other documents in the stack.Optionally, one or more sheets within a document may be orienteddifferently from one or more other sheets within the document, or fromdifferent documents. In some embodiments, a stack of documents cancomprise documents having the same or substantially the samecharacteristics. For example, the documents can be of the same material,shape, size, orientation and thickness. The documents and/or sheets ofthe documents may be of the same type.

The detection unit 102 can be used to identify one or more edges ofsingle sheets of documents comprising one or more of any number offlexible materials having information stored thereon. Identification ofthe single sheets of documents can facilitate separation of the sheetsfrom one another for subsequent processing, such as for scanning ofinformation from the individual sheets. In some embodiments, thedetection unit can be configured to identify a single sheet of paper,including printing paper, writing paper, and/or drawing paper. In someembodiments, the detection unit can be configured to identify a singlesheet of rice paper and/or thermal paper. The single sheet of paper maybe in a stack of documents, including a stack of papers or a stackincluding other types of flexible material. The single sheet may bewithin a document that may comprise one or more sheets that may befastened together. In some embodiments, one or more of the documents canbe other types of flexible material, including card stock, file folders,cardboard, acetate film, polyester film, synthetic paper, microfiche,tissue paper, X-ray film, blueprints, maps, cloth, parchment,combinations thereof and/or the like. In some embodiments, flexiblematerial comprises paper. A single document may comprise a single typeof flexible material or multiple types of flexible materials. Aplurality of documents may comprise as single type of flexible materialor multiple types of flexible materials. The flexible material may havea variety of sizes and/or shapes. In some embodiments, a target materialcan include a plurality of sheets of flexible material, where theflexible material comprises different sizes, shapes, thicknesses and/ororientations.

In some embodiments, the identification system 100 can be configured toidentify a sheet of flexible material having a thickness less than about0.3 millimeter (mm), 0.2 mm, 0.1 mm, 0.05 mm, or 0.1 mm. In someembodiments, the flexible material can have a thickness of up to about 5mm, about 4 mm, about 3 mm, about 2 mm. In some embodiments, theidentification system can be configured to identify a sheet of flexiblematerial having a thickness of about 0.01 mm to about 5 mm, includingabout 0.01 mm to about 4 mm, about 0.01 mm to about 3 mm, or about 0.01mm to about 4 mm. The identification system may be capable of detectingand/or differentiating sheets having thickness less than any of thevalues described herein, greater than any of the values describedherein, or falling within a range between any two of the valuesdescribed herein.

As described herein, in some embodiments, the flexible material may bepaper. The paper may be of any number of sizes, shapes and/orthicknesses. For example, the detection unit 102 may be configured toidentify edges of a sheet of paper having a thickness less than about0.3 mm, 0.2 mm, 0.1 mm, 0.05 mm, or 0.01 mm. In some embodiments, thepaper can have a thickness of up to about 5 mm, about 4 mm, about 3 mm,about 2 mm. In some embodiments, the identification system can beconfigured to identify a sheet of paper having a thickness of about 0.01mm to about 5 mm, including about 0.01 mm to about 4 mm, about 0.01 mmto about 3 mm, or about 0.01 mm to about 4 mm. The detection unit may becapable of detecting and/or differentiating sheets of papers havingthickness less than any of the values described herein, greater than anyof the values described herein, or falling within a range between anytwo of the values described herein. In some embodiments, the detectionunit can be configured to identify a single sheet of paper in a stack ofdocuments, including a sheet of paper in an orientation different froman immediately adjacent document in a stack, such as a document beneaththe paper.

In some embodiments, a document can be any number of different types ofproducts comprising one or more flexible material described herein. Insome embodiments, a document can be a foldable card, a business card, asheet having adhesive on at least a portion of one surface (e.g.,Post-It® Note note), an envelope, a folder, a document divider, and/or alabel. In some embodiments, the document can comprise any number ofsizes, including for example legal size, letter size, ledger, size, ortabloid size.

In some embodiments, the identification system 100, for example usinginformation collected by the detection unit 102, can be configured todetermine if any fastener is present on a document or stack ofdocuments. In some embodiments, the identification system can beconfigured to determine what type of fastener is on a document. Forexample, using information collected by the detection unit, theidentification system can be configured to determine if the fastener isone or more types of fasteners described herein. In some embodiments,the identification system can be configured to determine a location ofthe fastener on the document or document stack. For example, theidentification system can be configured to determine where on a documentor document stack one or more fasteners are located, including when adocument or document stack comprises multiple fasteners attached thereonon various locations on the document or document stack. In someembodiments, the identification system can be configured to determine anorientation of one or more fasteners on a document or document stack.For example, the identification system can be configured to determine alateral angle of a fasteners, and/or if a fastener is inverted. In someembodiments, the controller unit 104 can be configured to process theinformation collected by the detection unit to make one or moredeterminations described herein.

The identification system 100 may advantageously be configured to (1)identify single sheets of documents of various compositions and/ordimensions, and/or (2) identify information pertaining to any fasteners,such as the type, location and/or orientation of any fasteners on adocument or document stack. The identification system may be able toprocess documents that are not limited to a single type of material,orientation in which documents are positioned, and/or dimension, and mayhave one or more fasteners attached thereto in various manners.Pre-sorting documents processed by such a system may be reduced oreliminated, facilitating automated processing of documents without orsubstantially without manual intervention, thereby increasing speed ofdocument processing, reduce associated costs and/or improve reliabilityof the document process due at least in part to reduced human errors. Asdescribed herein, the identification system may be configured to processdocuments to prepare documents for a digitization process in whichinformation stored on the documents is extracted and stored digitally.An identification system which can be configured to handle documents ofvarious characteristics, including documents having different fastenersattached thereto in various manners, can facilitate digitization of theinformation on the documents, such that digital access of theinformation can be provided at reduced costs.

In some embodiments, the identification system 100 can be configured toidentify any number of different types of fasteners. For example, thedetection unit can be configured to differentiate between various typesof fasteners so as to determine which of the various types of fastenersis on a document or document stack, including the manner in which thefastener is attached to the document or document stack. A fastener maybe any number of different implements configured to hold multipledocuments together, such as various types of mechanical devicesconfigured to hold together multiple documents. For example, thefastener may comprise one or more portions which are configured to bepositioned over opposing surfaces of the documents the fastener isconfigured to hold together. In some embodiments, the fastener caninclude one or more portions which extend through a correspondingportion of each of the documents it is attached to. For example, thefastener may comprise one or more portions which pass through each ofthe documents it is configured to bind together and one or more otherportions which are configured to be positioned over opposing surfaces atop document and bottom document of the multiple documents bound by thefastener. The identification system can be configured to identifyfasteners such as paper clips, spring clips, metalbrads, punched holesfasteners, rubber bands, spiral binding, binder clips, spring binderclips, bookbinding, two-prong fasteners (e.g., two-prong metalfasteners), spiral rings, ring binding, gachuck clips, plastiklipplastic clips, corner clips, butterfly clamps, owl clips, circular paperclips, papercloops, adhesives (e.g., tape), combinations thereof and/orthe like. In some embodiments, the fasteners can comprise a file folder,including an accordion folder. In some embodiments, the detection unit102 can be configured to detect and/or identify staples, includingplastic staples, and non-ferrous metal staples. As described in furtherdetails herein, the interaction unit 106 can comprise one or more endeffectors configured to remove the identified fasteners.

In some embodiments, the detection unit 102 comprises one or morecomponents to sense an object, such as a document and/or fastener. Theone or more components may provide one or more of optical sensing,thermal sensing, laser imaging, infrared imaging, capacitance sensing,mass sensing, vibration sensing across at least a portion of theelectromagnetic spectrum, and magnetic induction sensing. In someembodiments, the detection unit comprises one or more tactilemodalities. The tactile sensing can be used to detect any anomalies on adocument or document stack. In some embodiments, the detection unit canbe configured to provide acoustic sensing, including sensing usingfrequencies beyond the range of human hearing. In some embodiments, theinformation collected by the detection unit can be used to determine ashape of a document, such as by detecting a boundary of the document. Insome embodiments, the detection unit can be used to determine a shape,material, orientation and/or location of one or more fasteners on adocument or stack of documents.

In some embodiments, the detection unit 102 can be configured to applyone or more techniques configured to detect metallic materials, such asto detect fasteners comprising a metallic material. In some embodiments,the detection unit can be configured to apply a magnetic field tofacilitate detection of ferromagnetic materials, including iron, nickel,cobalt, and/or combinations thereof.

The detection unit 102 can include a variety of sensing componentsconfigured to gather information relating to the documents to enableidentification of a single sheet of documents and/or the presence andtype of any fasteners present thereon. The detection unit can beconfigured to use various techniques to detect and/or identify a singlesheet of document and/or fastener. In some embodiments, the detectionunit can be configured to contact the document and/or fastener tomeasure one or more characteristics of the document and/or fastener,such as through one or more tactile modalities. In some embodiments, thedetection unit can be configured to collect information regarding thedocument and/or fastener without physically contacting the documentand/or fastener. In some embodiments, the detection unit can include oneor more of an emission source and a measurement device. In someembodiments, the detection unit can be configured to emit a detectionsignal configured to probe the document and/or fastener, including forexample an electromagnetic signal and/or an acoustic signal. In someembodiments, the detection unit comprises both an emission source and ameasurement device. For example, the measurement device may beconfigured to measure an electromagnetic radiation signal emitted byand/or reflected by the document and/or fastener, due at least in partto response of the document and/or fastener to the detection signalemitted by the emission source. As described in further details herein,the detection unit may comprise one or more components for a photometricstereovision system. In some embodiments, the detection unit can includea measurement device to collect information regarding the documentand/or fastener, without or substantially without emitting any detectionsignals for the measurement. For example, as described in furtherdetails herein, the detection unit may comprise one or more componentsfor an infrared sensor and/or an inductive sensor.

In some embodiments, the detection unit 102 may comprise an illuminationsource configured to illuminate the document or document stack with oneor more types of electromagnetic radiation. In some embodiments, theelectromagnetic radiation can include illumination in one or more of thevisible spectrum, infrared spectrum, the ultraviolet spectrum, andionizing radiation spectrum. In some embodiments, the ionizing radiationcan include x-rays.

In the some embodiments, the detection unit 102 may be configured toilluminate a document or stack of documents for analysis usingphotometric stereovision. The detection unit may be configured toprovide information to enable determining three-dimensional (3-D)information relating to the target material. In some embodiments, thedetection unit may be configured to obtain information to determine aboundary of a sheet of document to facilitate identification of a singlesheet of document. In some embodiments, the detection unit can beconfigured to provide edge detection of a sheet of document.

Referring again to FIG. 1, the interaction unit 106 may be configured toperform various tasks to manipulate a document, document stack and/or afastener. The interaction unit may comprise one or more tools configuredto perform such tasks. In some embodiments, the interaction unit maycomprise one or more positioning components for coupling to the one ormore tools, for example to facilitate moving the one or more tools to adesired location and/or orientation. For example, the one or morepositioning components may comprise one or more mechanical arms forbringing the one or more tools to the document, document stack and/orfastener, and/or orienting the one or more tools relative to thedocument, document stack and/or fastener. In some embodiments, the oneor more components may comprise one or more robot arms. The one or morepositioning components can be coupled, such as releasably coupled, toappropriate tools for performing desired tasks. The tools may beinterchangeable. For example, the one or more positioning components maybe configured to be switch between different tools depending on the taskdesired. The tools may be exchanged, swapped, detached, and/orreattached to the one or more positioning components. In someembodiments, the tools are not interchangeable.

The interaction unit 106 may comprise one or more tools to move adocument and/or document stack, such as to transfer the document and/ordocument stack between a first location and a second location. In someembodiments, the interaction unit may comprise one or more tools tochange an orientation of a document and/or document stack, such as torotate the document and/or document stack. In some embodiments, theinteraction unit may comprise a tool to flip over a document and/ordocument stack. In some embodiments, the interaction unit may compriseone or more tools to separate a document or document stack from anotherdocument and/or document stack. In some embodiments, the interactionunit may comprise one or more tools for manipulating a fastener, such asto remove the fastener from a document and/or document stack.

In some embodiments, the interaction unit 106 may be configured toseparate a document or document stack from another document and/ordocument stack using various techniques, including using guidance fromthe detection unit 102. The interaction unit may be configured toseparate a document or document stack from another document and/ordocument stack and into groups, including by folder (e.g., by filefolder, and/or accordion folder), attached and/or bound groups, and/orinto single sheets. In some embodiments, the interaction unit can beconfigured to separate a document or stack of documents from one or morefolders, including opening of folders (e.g., accordion folders and/orfile folders) and/or removal of the document or document stack fromfolders. In some embodiments, the interaction unit can be configured toperform the separation by physically contacting the document or documentstack. In some embodiments, the interaction unit can be configured toperform the separation by using air pressure, such as using air pressuregreater than and/or less than atmospheric pressure. In some embodiments,the interaction unit can be configured to perform the separation byusing gravity, such as by placing the document or document stack on asloped plane to enable gravity to separate the document or documentstack from any other documents and/or stacks of documents. In someembodiments, the interaction unit can comprise a mechanical roller canbe used to facilitate separation using gravity. For example, themechanical roller can be used to contact the document or document stackto facilitate the separation.

In some embodiments, one or more positioning components of theinteraction unit 106 may comprise one or more robot arms. The robot armcan be a multi-axis robot arm. In some embodiments, the robot arm canhave two or more axes of motion. In some embodiments, the robot arm canbe a two-axis robot arm, a three-axis robot arm, four-axis robot arm, afive-axis robot arm or a six-axis robot arm. In some embodiments, robotarm may be a single-axis robot arm.

In some embodiments, the one or more tools of the interaction unit 106may comprise an end effector configured to be coupled to a positioncomponent, such as a robot arm. In some embodiments, the robot arm canbe configured to be coupled to one end effector at any one time. In someembodiments, the robot arm can be configured to be coupled to more thanone end effector at any one time. In some embodiments, the robot arm canbe configured to switch between different end effectors. For example,the robot arm can be configured to release an end effector once a taskusing the end effector is completed and to couple to a different endeffector suited to perform a subsequent task. In some embodiments, theentire robot arm can be switched out.

In some embodiments, one or more end effectors can be coupled to aposition component, such as a robot arm, to perform various tasks. Insome embodiments, one or more end effectors can be configured forfastener removal. In some embodiments, one or more end effectors can beconfigured for document manipulation, such as to manipulate individualsheets of documents and/or multiple sheets of documents. In someembodiments, one or more end effectors can be configured to manipulateone or more fasteners and one or more documents. The one or more endeffectors can be releasably coupled to the robot arm. For example, therobot arm can be configured to switch between appropriate removal toolsbased upon the detected type of fastener.

In some embodiments, more than one robot arm can be used. In someembodiments, a robot arm for fastener removal can be different from arobot arm configured for manipulation of sheets of documents.

In some embodiments, the identification system 100 may be configured toprovide documents for further processing, such as providing documentshaving fasteners removed for digitization of information stored on thedocuments. In some embodiments, the processed documents provided by theidentification system can be scanned such that information storedthereon can be extracted. For example, information on the documents maybe extracted using one or more optical scanning techniques. One or morecharacter recognition algorithms can be applied to convert theinformation into digital format. In some embodiments, information fromthe documents can be extracted using an optical character recognition(OCR) process.

The identification system 100 can enable fully automated identificationof single sheets of documents, presence and types of fasteners, and/orremoval of fasteners. The identification system can be a part of anassembly line configured to enable automated digitization of informationstored on physical documents. In some embodiments, each of the detectionunit 102 and the interaction unit 106 can be configured to operatewithout or substantially without human interaction. The detection unitand the interaction unit may be configured to operate together to fullyautomate identification of documents and/or fasteners, and/or removal offasteners from documents presented to the identification system. Forexample, the system can be configured to process a document or documentstack, such as to identify sheets of documents, presence and types offasteners, and/or removal of fasteners, after receiving input from anoperator to initiate the process. In some embodiments, theidentification system can be semi-automated. For example, theidentification system can be configured to receive input from anoperator after processing of a document or document stack is initiated,such as at one or more points during the process. The operator may beprompted by the system during the process to input a confirmationcommand, and/or one or more parameters regarding the document ordocument process being processed. The identification system 100 can bepart of an assembly line configured to fully automate or semi-automatedigitization of information stored on the document or document stack.

The controller unit 104 can be configured to control one or more othercomponents of the identification system 100, such as to enableautomation of processes to identify sheets of documents, presence andtypes of fasteners, and/or removal of fasteners. The controller unit 104may comprise one or more processors, including a central processing unit(CPU). In some embodiments, the processor may comprise a single coreprocessor. In some embodiments, the processor may comprise a multi-coreprocessor. In some embodiments, the controller unit comprises aplurality of processors for parallel processing. The controller unit mayhave one or more of a memory (e.g., random-access memory, read-onlymemory, and/or flash memory) and an electronic storage unit (e.g., harddisk). The electronic storage unit can be a data storage unit and/ordata repository for storing data. In some embodiments, the controllerunit can have a communication interface (e.g., network adapter) forcommunicating with one or more other components and/or systems. Forexample, as described in further details herein, the controller unit maybe configured to communicate with one or more other components of theidentification system 100, such as the detection unit 102 and/or theinteraction unit 106. In some embodiments, the controller unit can be incommunication with one or more remote systems, such as remote serverslocated in a different room, different building, and/or differentfacilitate. In some embodiments, the controller unit can have one ormore peripheral devices, such as cache, other memory, data storageand/or electronic display adapters. The memory, storage unit, interfaceand/or peripheral devices can be in communication with the processor ofthe controller unit through a communication bus, such as a motherboard.

The controller unit 104, such as a processor and a memory of thecontroller unit, can be in communication with the detection unit 102and/or the interaction unit 106. The controller unit can be configuredreceive information from the detection unit, such as informationcollected by the detection unit relating to the document or documentstack being processed. The controller unit memory may be configured tostore information received from the detection unit and/or variousalgorithms for performing the analyses described herein. In someembodiments, the controller unit can be configured to process theinformation from the detection unit relating to the document or documentstack using the one or more algorithms. For example, the processor ofthe controller unit can execute a sequence of machine-readableinstructions, which can be embodied in a program or software. Theinstructions may be stored in a memory location, such as the memory ofthe controller unit. The instructions can be directed to the processor,which can subsequently program or otherwise configure the processor toimplement methods of the present disclosure. Examples of operationsperformed by the processor can include fetch, decode, execute, and/orwriteback. In some embodiments, the processor can be configured toexecute the various algorithms using the information relating to thedocument, document stack and/or fastener to perform the analysis asdescribed herein.

In some embodiments, the controller unit can be in communication withone or more remote servers. In some embodiments, one or more of thevarious analyses of information from the detection unit 102 can beperformed remotely, and the analysis is transmitted to the controllerunit 104. The controller unit can be operatively coupled to a computernetwork (“network”) with the aid of the communication interface, such asto a wired and/or wireless network. The network can be the Internet, aninternet and/or extranet, or an intranet and/or extranet that is incommunication with the Internet. The network in some cases is atelecommunication and/or data network. In some embodiments, the networkcomprises a local area network (“LAN”), and/or a wide area network(“WAN”). The network can include one or more computer servers, which canenable distributed computing, such as cloud computing. The network, insome cases with the aid of the computer system, can implement apeer-to-peer network, which may enable devices coupled to the computersystem to behave as a client or a server.

The controller unit 104 can be configured to send instructions, based onthe analyses, to the detection unit and/or the interaction unit tocontrol movement of the detection unit 102 and/or interaction unit 106,so as to enable automation of the identification and manipulationprocess as described herein, such as to enable a fully automated orsemi-automated process.

In some embodiments, the processor of the controller unit 104 can bepart of a circuit, such as an integrated circuit. One or more othercomponents of the controller unit can be included in the circuit. Insome cases, the circuit is an application specific integrated circuit(ASIC). In some embodiments, the storage unit of the controller unit canstore files, such as drivers, libraries and saved programs. Thecontroller unit can store user data, e.g., user preferences and userprograms. The controller unit in some cases can include one or moreadditional data storage units that are external to the controller unit,such as located on a remote server that is in communication with thecontroller unit through an intranet or the Internet. The controller unitcan communicate with one or more remote computer systems through thenetwork as described herein. For instance, the controller unit cancommunicate with a remote computer system of a user. Examples of remotecomputer systems include personal computers (e.g., portable PC), slateor tablet PC's (e.g., Apple® iPad, Samsung® Galaxy Tab), telephones,Smart phones (e.g., Apple® iPhone, Android-enabled device, Blackberry®),or personal digital assistants. In some embodiments, an operator canaccess the controller unit via the network.

Methods as described herein can be implemented by way of machine (e.g.,computer processor) executable code stored on an electronic storagelocation of the controller unit 104, such as, for example, on the memoryor electronic storage unit of the controller unit. The machineexecutable or machine readable code can be provided in the form ofsoftware. During use, the code can be executed by the processor of thecontroller unit. In some cases, the code can be retrieved from thestorage unit and stored on the memory for ready access by the processor.In some situations, the electronic storage unit can be precluded, andmachine-executable instructions are stored on memory.

The code can be pre-compiled and configured for use with a machinehaving a processer adapted to execute the code, or can be compiledduring runtime. The code can be supplied in a programming language thatcan be selected to enable the code to execute in a pre-compiled oras-compiled fashion.

One or more aspects of the systems and methods provided herein, such asthe controller unit 104, can be embodied in programming. Various aspectsof the technology may be thought of as “products” or “articles ofmanufacture” typically in the form of machine (or processor) executablecode and/or associated data that is carried on or embodied in a type ofmachine readable medium. Machine-executable code can be stored on anelectronic storage unit of the controller unit, such as memory (e.g.,read-only memory, random-access memory, flash memory) or a hard disk.“Storage” type media can include any or all of the tangible memory ofthe computers, processors or the like, or associated modules thereof,such as various semiconductor memories, tape drives, disk drives and thelike, which may provide non-transitory storage at any time for thesoftware programming. All or portions of the software may at times becommunicated through the Internet or various other telecommunicationnetworks. Such communications, for example, may enable loading of thesoftware from one computer or processor into another, for example, froma management server or host computer into the computer platform of anapplication server. Thus, another type of media that may bear thesoftware elements includes optical, electrical and electromagneticwaves, such as used across physical interfaces between local devices,through wired and optical landline networks and over various air-links.The physical elements that carry such waves, such as wired or wirelesslinks, optical links or the like, also may be considered as mediabearing the software. As used herein, unless restricted tonon-transitory, tangible “storage” media, terms such as computer ormachine “readable medium” refer to any medium that participates inproviding instructions to a processor for execution.

Hence, a machine readable medium, such as computer-executable code, maytake many forms, including but not limited to, a tangible storagemedium, a carrier wave medium or physical transmission medium.Non-volatile storage media include, for example, optical or magneticdisks, such as any of the storage devices in any computer(s) or thelike, such as may be used to implement the databases, etc. shown in thedrawings. Volatile storage media include dynamic memory, such as mainmemory of such a computer platform. Tangible transmission media includecoaxial cables; copper wire and fiber optics, including the wires thatcomprise a bus within a computer system. Carrier-wave transmission mediamay take the form of electric or electromagnetic signals, or acoustic orlight waves such as those generated during radio frequency (RF) andinfrared (IR) data communications. Common forms of computer-readablemedia therefore include for example: a floppy disk, a flexible disk,hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD orDVD-ROM, any other optical medium, punch cards paper tape, any otherphysical storage medium with patterns of holes, a RAM, a ROM, a PROM andEPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wavetransporting data or instructions, cables or links transporting such acarrier wave, or any other medium from which a computer may readprogramming code and/or data. Many of these forms of computer readablemedia may be involved in carrying one or more sequences of one or moreinstructions to a processor for execution.

The controller unit 104 can include or be in communication with a userinterface (UI). Examples of UI's include, without limitation, agraphical user interface (GUI) and web-based user interface. Theoperator may be able to input information for controlling the systemusing the user interface. For example, the operator may be able to enterone or more parameters using the user interface, to initiate a processand/or as part of the process, including when prompted by the system.The user interface may comprise an electronic display. The electronicdisplay may be configured to allow an operator to view informationrelating to one or more components of the identification system 100,and/or one or more parameters of a process previously, currently beingand/or will be completed by the system. The electronic display mayenable the operator to view information gathered by the detection unit102.

As described herein, one or more systems and/or processes as describedherein can enable a fully automated and/or semi-automated process forprocessing documents and/or document stacks, such as to identify sheetsof documents, presence and types of fasteners, and/or removal offasteners. In some embodiments, automating the identification ofdocuments and/or fasteners, and/or removal of the fasteners canadvantageously provide increased throughput in the processing ofdocuments to digitize information stored thereon.

Although the detection unit 102, controller unit 104, and interactionunit 106 are described with reference to FIG. 1 as being distinctcomponents of the identification system 100, it will be understood that,in some embodiments, one or more of the detection unit 102, controllerunit 104, and interaction unit can be a part of one component.

In some embodiments, one or more of a detection unit and an interactionunit may be a part of different systems. For example, the detection unitcan be a part of a first automated system separate from and/or locatedat a different location from a second automated system comprising theinteraction unit, and documents processed by the first automated systemmay be transferred to the second automated system for furtherprocessing.

FIG. 2 is a schematic diagram of an example of a document and fasteneridentification system 200. The document and fastener identificationsystem 200 can be an example of the identification system 100 asdescribed with reference to FIG. 1. The document and fasteneridentification system 200 can include an illumination source 210, animage capture device 230, a robot arm 240, a document pedestal 250, afirst document platform 260, and a second document platform 270. Thedocument and fastener identification system 200 may include a controller202 in electrical communication with one or more components of thedocument and fastener identification system 200, such as theillumination source 210, image capture device 230, robot arm 240,document pedestal 250, first document manipulator 260, and/or seconddocument manipulator 270, to control the one or more components and/orreceive information from the one or more components for controlling thecomponents. The controller 202 may include a processor for receiving andprocessing the image data from the image capture device 230.

The document and fastener identification system 200 may include a userinterface 204 to allow an operator to interact with the system. Forexample, the operator may be able to manually control one or morecomponents of the system and/or input information for controlling thesystem using the user interface. The user interface can allow theoperator to interact with the system. The operator may input valuesusing the user interface to initiate and/or control a process. In someembodiments, the operator may input one or more values using the userinterface such that the system can operate autonomously. For example, adocument, document stack, multiple documents, and/or multiple documentstacks can be loaded onto the system (e.g., by an operator and/or anautomated process), such as onto an input port of the system, and thesystem may be configured to process the document without furtheroperator input. In some embodiments, the operator may input one or moreparameters to initiate the process and the system can be configured tocomplete the process fully automated without further operatorintervention. In some embodiments, the system can be configured toreceive a start instruction and the system can complete the processwithout further instructions from the operator. For example, upon theoperator entering a “start” command, the system can proceed toilluminate a document or document stack using the illumination source210, capture a plurality of images of a document or document stack usingthe image capture device 230, and manipulate the document or documentstack using one or more of the robot arm 240, first document manipulator260, and second document manipulator 270. In some embodiments, thesystem can be configured to process the document, document stack,multiple documents, and/or multiple document stacks based upon one ormore parameters entered by the operator, such as one or more parametersrelating to one or more characteristics of the document, document stack,multiple documents, and/or multiple document stacks.

In some embodiments, the system 200 can be configured to besemi-autonomous such that the system is configured to receive input fromthe operator at one or more points during the operation of the systemafter initiation. For example, the system 200 can be configured toprompt the operator for input to input one or more characteristics ofthe document, document stack, multiple documents, and/or multipledocument stacks being processed, and/or confirm an operation and/ordetected characteristic of the document, document stack, multipledocuments, and/or multiple document stacks. For example, an operator canbe requested to input a “confirm” command during one or more steps ofthe process before the system will proceed further. In some embodiments,the system can be configured to operate both in a fully automated modeand a semi-automated mode, such as based on input from an operator. Insome embodiments, the system can be configured to allow the operator tointervene during the process, such as by inputting a “pause” and/or“stop” command.

In some embodiments, the operator can manually control one or morecomponents of the system using the user interface, such as one or moreof the illumination source 210, image capture device 230, robot arm 240,document pedestal 250, first document platform 260, and second documentplatform 270. In some embodiments, the user interface may be co-locatedwith the system. In some embodiments, the user interface may be remotefrom the system, for example to enable remote operation of the systemand/or remote viewing of one or more parameters of the process performedby the system. In some embodiments, the user interface can be in adifferent room, building, and/or facility. In some embodiments, the userinterface can be in the same room, building, and/or facility.

In some embodiments, the user interface 204 can include a visual display206. In some embodiments, the visual display can include a display ofvarious parameters of the system 200, including one or more parametersof the document and/or document stack being processed. As will bedescribed in further details herein, the visual play can be configuredto display to an operator an unprocessed image, such as an imagecaptured by the image capture device 230 and/or a processed image, suchas a processed image provided by the processor of the controller 202using information from the unprocessed image. In some embodiments, thevisual display may display one or more status indicators of thecomponents of the system, including status indicators of one or more ofthe illumination source 210, image capture device 230, robot arm 240,document pedestal 250, first document platform 260, and second documentplatform 270. The visual display may be configured to display to theoperator values entered by the operator or a previous operator. Thevisual display may show a process history of the system.

In some embodiments, the document and fastener identification system 200can be configured to identify one or more edges of a single documentand/or presence of a fastener on a document or document stack at leastin part by analyzing any shadows captured by the document and fasteneridentification system 200. In some embodiments, the document andfastener identification system 200 can be configured to providephotometric stereovision analysis of the document or document stack. Theanalysis may be used to identify edges of a single document and/orinformation about a fastener, if present.

The document pedestal 250 can be configured to receive a document ordocument stack analyzed by the document and fastener identificationsystem 200. For example, a document or document stack can be received onan upper surface 252 of the document pedestal 250. The document ordocument stack can be positioned on the document pedestal manually, suchas by an operator. In some embodiments, the document or document stackcan be positioned on the document pedestal by an automated process. Forexample, the document or document stack can be presented to the documentand fastener identification system 200 at an input, and the document ordocument stack can be positioned onto the document pedestal through anautomated process. The document or stack of documents may be presentedas a single sheet of document, a plurality of sheets of documents, asingle stack of documents, a plurality of stacks of documents.

The document or document stack can be presented to the system 200 invarious forms. In some embodiments, the system 200 can be configured toreceive the document or document stack in a container, such as a tray, abox, a bin, and/or a folder. For example, a container containing adocument stack can be loaded onto a receiving port of the identificationsystem and the container can be moved by the system to a desired heightto transfer one or more of the documents from within the container tothe pedestal 250. The system may comprise one or more mechanicalcomponents to move the container vertically and/or laterally, such asvia tracks. For example, the container can be lifted, lowered and/ormoved laterally by the system to reach a desired position fortransferring documents from the container to the pedestal 250. Once thedesired position is reached, the system may be configured to automatetransfer of documents from the container. As described herein, a desiredheight of the container can be preset and/or determined using one ormore sensors.

The illumination source 210 can be positioned relative to the documentpedestal 250 such that the document or document stack positioned on thepedestal 250 can be illuminated by the illumination source 210. Theillumination source may be positioned above the document pedestal 250and oriented such that the illumination from the illumination sourceprovides desired illumination of the document or document stack on thepedestal. For example, the illumination source can be oriented such thatone or more illumination devices providing the illumination face anupper surface 252 surface of the pedestal. In some embodiments, asurface of the illumination devices through which illumination isemitted are oriented at an angle of about 30° to about 90° relative to aupper surface of the pedestal, including at an angle of about 30° toabout 80°, about 30° to about 60°, about 45° to about 90°, or about 45°to about 60°. The orientation angle of the illumination source may beless than any of the values described herein, greater than any of thevalues described herein, or falling within a range between any two ofthe values described herein. The illumination source 210 can bepositioned relative to the pedestal 250 such that other components ofthe system 200 do not cast any shadow on any portion of the document ordocument stack positioned on the pedestal 250 when the document ordocument stack is illuminated by the illumination source 210. A shadowcast upon the document or document stack may result in a false positivein the identification of a fastener and/or one or more edges of adocument.

The illumination source 210 can comprise a plurality of illuminationmodules. The plurality of illumination modules may each be positioned ata different location relative to the pedestal 250 such that the documentor document stack on the pedestal 250 can be illuminated by sourceslocated at different positions relative to the document or documentstack. For example, the illumination modules can be at differentlocations relative to one another. Illuminating the document or documentstack from different positions can provide allow capture of differentshadow profiles of three-dimensional features, such as the edges of asingle document and/or any three-dimensional objects protruding from thetop of the document or document stack, such as a fastener.

In some embodiments, the illumination source 210 can comprise fourillumination modules 212, 214, 216 and 218. The four illuminationmodules can positioned over and surround the pedestal 250. For example,one of the illumination modules can be at each of a 0° position, 90°position, 180° position and 270° position. In some embodiments, the fourillumination modules can be positioned differently around the pedestal.The four illumination modules can be equidistant or substantiallyequidistant from the pedestal to facilitate illuminating the document ordocument stack by the same or similar illumination intensity by each ofthe four illumination modules. The four illumination modules can be at avariety of vertical distances from the upper surface 252 of thepedestal. The vertical distance can selected such that the document ordocument stack on the pedestal is sufficiently illuminated by theillumination modules. In some embodiments, a vertical distance of one ormore of the illumination modules from the upper surface 252 of thepedestal 250 can be about 4 feet (ft) to about 10 ft, including about 4ft to about 8 ft, or about 4 ft to about 6 ft. The vertical distance ofone or more of the illumination modules from the upper surface of thepedestal may be less than any of the values described herein, greaterthan any of the values described herein, or falling within a rangebetween any two of the values described herein.

In some embodiments, a different number of illumination modules can beused. In some embodiments, fewer than four illumination modules can beused, such as two or three. In some embodiments, more than fourillumination modules can be used, such as five, six, seven, eight, ormore. The plurality of illumination modules may be positioned such thatthey modules are uniformly or substantially uniformly distributedradially over the pedestal 250. An illumination source comprising nnumber of illumination modules may have the modules at each of amultiple of 360°/n position around the pedestal. For example, a systemcomprising five illumination modules may comprise a module at each of a0° position, 72° position, 144° position, 216° position, and 288°position.

As described herein, the illumination source 210 may be positioned abovethe document pedestal 250 and oriented such that the illumination fromthe illumination source provides desired illumination of the document ordocument stack on the pedestal. For example, each of the plurality ofillumination modules of an illumination source can have an illuminationemitting surface oriented at an angle of about 30° to about 90° relativeto a upper surface 252 of the pedestal 250, including at an angle ofabout 30° to about 80°, about 30° to about 60°, about 45° to about 90°,or about 45° to about 60°. The orientation angle of the illuminationmodules may be less than any of the values described herein, greaterthan any of the values described herein, or falling within a rangebetween any two of the values described herein.

Each of the plurality of illumination modules can comprise one or moreillumination devices oriented toward the document or document stack forilluminating the document or document stack. In some embodiments, eachof the plurality of illumination modules can comprise one or morelight-emitting diodes (LEDs). The LEDs may be configured to emit lightin the visible range. In some embodiments, the illumination modules maycomprise one or more of bar lights, ring lights, and row of lights. Insome embodiments, the illumination modules may comprise illuminationdevices for primary and secondary optics measurements. In someembodiments, LEDs can be configured to emit light outside of the visiblerange (e.g., electromagnetic radiation having a wavelength in the rangeof about 400 nanometers (nm) to about 700 nm). In some embodiments, theillumination modules can comprise one or more of a fluorescentillumination, quartz halogen illumination, metal halide illumination,xenon illumination, and/or high pressure sodium illumination.

The plurality of illumination modules may be illuminated individually insequence such that one or more images of the document or document stackcan be captured when each individual illumination module is illuminated.The illumination modules may be illuminated in any sequence. In someembodiments, the illumination modules may be illuminated in a clockwiseorder. In some embodiments, the illumination modules may be illuminationin a counterclockwise order. Sequential illumination of the modules maybe manually set and/or automatically selected, such as based on one ormore system parameters and/or characteristics of the documents beingprocessed. The rate at which the illumination modules are activated insequence can be selected to increase throughput of the system 200 whileallowing sufficient time for the image capturing device 230 to image thedocument or document stack. The rate at which the illumination modulesare illuminated may be less than any of the values described herein,greater than any of the values described herein, or falling within arange between any two of the values described herein. In someembodiments, the rate at which the illumination modules are illuminatedis faster than rate which can be perceived by the human eye. The rate atwhich the illumination modules are activated can be selected based onone or more parameters of the image capture device 240, such as thespeed at which the image capture device can capture an image, includingfor example an exposure time of the image capture device. For example,the rate of illumination can be selected based on the exposure time of adigital camera. The rate of illumination can be selected to be at leastas long as the exposure time of the digital camera. In some embodiments,each illumination modules can be illuminated for a duration of greaterthan about 0.1 milli-seconds (msec), about 0.2 msec, about 0.5 msec,about 1 msec, about 5 msec, or about 10 msec. In some embodiments, eachillumination module can be illuminated for a duration of about 0.1 msecto about 20 sec, about 1 sec to about 15 sec, about 5 sec to about 10sec.

The image capturing device 230 can be positioned above the pedestal 250such that images of the document or document stack on the pedestal 250can be captured without any obstructions. In some embodiments, the imagecapturing device 230 can be positioned such that its shadow is not castupon the document or document stack when any of the illuminationmodules, such as the illumination modules 212, 214, 216 and 218 areactivated. In some embodiments, the image capturing device 230 ispositioned above each of the illumination modules. In some embodiments,the image capturing device 230 is equidistant or substantiallyequidistant to each of the illumination modules.

In some embodiments, the image capture device 230 is a digital camera.In some embodiments, the image capture device 230 is a 5-megapixeldigital camera. The camera can have resolution values other than 5megapixels. The camera can have lower or higher resolution, such asabout 3-megapixels, about 8-megapixels, or about 12 megapixels. Theresolution of the digital camera can be selected to provide capture ofdesired features on the document or document stack being processed. Insome embodiments, the image capturing device 230 can be another type ofimaging device selected based on the type of illumination source 210used.

The image capture device 230 can be configured to capture one or moreimages of a document or document stack on the pedestal 250 each time thedocument or document stack is being illuminated by each of a pluralityof illumination modules, such as the plurality of illumination modules212, 214, 216 and 218. As described herein, the plurality ofillumination modules positioned at different locations can be activatedin sequence such that images of the document or document stack can becaptured while being illuminated from different angles. The capturedimages can be transmitted from the image capture device to thecontroller 202 for analysis, for example such that a boundary of adocument, and/or one or more features of a fastener can be identified.

As described herein, the controller 202 may comprise a memory configuredto store the information from the image capture device 210 and/or one ormore algorithms used to process the information. For example, machinereadable instructions for executing processing of the information may bestored in the memory. The controller may comprise a processor configuredto process the captured images from the capture device, for example toexecute a sequence of machine-readable instructions to analyze thecaptured images.

In some embodiments, the analysis performed by the controller comprisescomparing the images of the document or document stack captured by thecapture device to identify any shadows in the images. For example, thecontroller can be configured to perform a comparison between the imagesto detect any changes in the images, such as a pixel-by-pixelcomparison, to identify the presence of any shadows. Shadows may be castby one or more edges of a document or document stack onto the uppersurface 252 of the pedestal, and/or onto one or more documents and/ordocument stacks beneath the document or document stack being processed.Shadows may be cast by one or more fasteners on the document or documentstack. For example, the controller can be configured to generate ashadow profile of the document or document stack using the capturedimages.

In some embodiments, the controller 202 can be configured to determineone or more characteristics of the document, document stack and/or anyfasteners on the document or document stack based on the shadow profile.A size, shape, orientation, and/or thickness of a document or documentstack can be determined using the shadow profile. In some embodiments,the shadow profile can be used to identify a boundary of a document,such as a single sheet of paper. In some embodiments, the shadow profilecan be used to determine the type of fastener on the document ordocument stack being processed. In some embodiments, the shadow profilecan be used to determine a location and/or orientation of the fasteneron the document or document stack.

The controller 202 can be configured to make one or more determinationsregarding the document, document stack and/or fastener using one or morealgorithms stored therein, including one or more pattern recognitionalgorithms. For example, the shadow profiles generated by the controllercan be compared to one or more known and/or reference patterns todetermine a size, shape, orientation, and/or thickness of a document,and/or a type, location and/or orientation of a fastener. In someembodiments, the controller can be configured to use the shadow profilesto determine which features in an image are three-dimensional features.In some embodiments, the controller can be configured to determine whichof the three-dimensional features present correspond to a boundary of adocument and/or an actual fastener on the document or document stack.The controller may be configured to disregard the three-dimensionalfeatures which are not actual documents and/or fasteners. For example,the controller can be configured to process information gathered by theimage capture device 230 using one or more algorithms stored in a memoryof the controller to determine whether shadows present in the images areshadows cast by features of a document of interest and/or an actualfastener. The controller can generate instruction signals forcontrolling one or more components of the system to manipulate thedocument or document stack and/or fastener to remove the fastener basedon the analysis of which three-dimensional features are features of aboundary of a document, and/or features of an actual fastener. Thecontroller may use one or more pattern recognition algorithms to performone or more analyses described herein. In some embodiments, shadowspresent in the images captured by the image capture device can becompared to one or more thresholds and/or references, including knownreference patterns, such as known reference patterns of known fastenersand/or portions of fasteners, and/or patterns of flexible materialand/or portions of flexible material. The controller can be configuredto determine whether an actual fastener is present, boundary of thedocument, and/or orientation and/or position of the document and/orfastener, using the comparison. In some embodiments, one or more of theanalysis is performed by a controller remote from the system 200 andthat analysis which is performed remotely is transmitted to thecontroller 202 for controlling one or more components of the system,such by one or more wired and/or wireless networks as described herein.For example, the remote controller can at a different location in thefacility at which the system 200 is located, or at a facility differentfrom that of the system 200.

As described in further details herein, the controller 202 can beconfigured to send instructions to one or more of the robot arm 240,document pedestal 250, first document platform 260, and second documentplatform 270 to manipulate the document or document stack and/or anyfasteners on the document or document stack.

Referring again to FIG. 2, as described herein, the system 200 caninclude a robot arm 240 comprising one or more tools (not shown)configured to manipulate the document or document stack and/or afastener, such as one or more end effectors. The one more end effectorscan be configured to be detachably coupled to the robot arm, such as ata distal portion 242 of the robot arm. The controller 202 can beconfigured to send instructions to the robot arm to control movement ofthe robot arm and/or the one or more end effectors coupled to the robotarm. For example, the controller can be configured to instruct the robotarm to attach to one or more end effectors appropriate for a desiredtask. The desired task can be selected based at least in part on thedetermination made regarding the size, shape, orientation, and/orthickness of a document or document stack, and/or a type, locationand/or orientation of any fasteners on the document or document stack,as described herein. After the task is completed, the controller caninstruct the robot arm to release the one or more end effectors, such asto place the one or more end effectors in a storage component of thesystem.

As will be described in further details herein, in some embodiments, therobot arm 240 can be configured to be releasably coupled to a documenttransfer end effector and/or a fastener removal end effector. Thecontroller 202 can be configured to send instructions to the robot armregarding which end effector to use based on the determination maderegarding the size, shape, orientation, and/or thickness of a documentor document stack, and/or a type, location and/or orientation of anyfasteners on the document or document stack.

The system 200 can comprise one or more end effector storage componentsconfigured to store the end effectors. End effectors not being used bythe robot arm 240 may be stored. For example, the robot arm 240 mayperform a task using a first end effector and subsequently, aftercompletion of the task, move to the end effector storage to release thefirst end effector, such as positioning the first end effector at apreset location within the end effector storage. The robot arm may thenretrieve a second end effector from the end effector storage to performa second task. The first and second end effectors may comprise one ormore end effectors configured to manipulate a document, document stack,and/or fastener. The robot arm can subsequently return the second endeffector to the storage once the second task has been completed. Thisprocess can be repeated until the desired manipulation of the documentor document stack is completed, such as the removal of all fastenersfrom the document or document stack. For example, the robot arm mayfirst retrieve a document transfer end effector to separate the documentor document from other document(s) and/or document stack(s) on thepedestal 250. After separation is complete, the robot arm can beinstructed by the controller 202 to return the document transfer endeffector to the storage area. The controller may then instruct the robotarm to couple to a fastener removal end effector to remove a fastenerfrom the document or document stack. The fastener removal end effectorcan be selected based on the information collected regarding thefastener. The controller can instruct the robot arm to return thefastener removal end effector after the fastener removal step iscompleted.

In some embodiments, the controller 202 can be configured to control oneor more of the first document platform 260 and the second documentplatform 270 for manipulating a document or document stack beingprocessed by the system 200. The first and/or second platforms can beused in combination with a robot arm 240. In some embodiments, thecontroller can be configured to instruct the first document platform andsecond document platform to lift and/or separate the document ordocument stack from any other document(s) or document stack(s) on thepedestal 250, such as in combination with one or more end effectorscoupled to the robot arm. For example, an end effector coupled to therobot arm can be used to lift a document or document stack from thepedestal, such as by using a suction force placed over at least aportion of the document or document stack. One of the first documentplatform or the second document platform can be moved to be subsequentlypositioned underneath the document or document stack lifted by the endeffector on the robot arm. In some embodiments, positioning of adocument platform under the document or document stack may facilitateseparation of the document or document stack from any other document(s)or document stack(s) on the pedestal. In some embodiments, the documentor document stack can remain on the first document platform when afastener attached thereto is removed. For example, the document ordocument stack can remain on the first document platform while afastener is removed using one or more end effectors coupled to the robotarm.

In some embodiments, both the first document platform 260 or the seconddocument platform 270 can be used to manipulate the document or documentstack on the pedestal 250, such as to flip the document or documentstack over. The document or document stack may be flipped over tofacilitate removal of one or more fasteners attached thereto. A fasteneron the document or documents stack, such as a staple, may be in aninverted position. An inverted staple may have a bridging portion (e.g.,a crown) between its two leg portions on a bottom surface of thedocument or document stack. For example, the bridging portion may beoriented towards the upper surface 252 of the pedestal 250 while its twoleg portions are oriented away from the upper surface, such as towardsthe illumination source 210 above the pedestal. The document or documentstack may be flipper over to facilitate removal of the inverted staple,such that the bridging portion between its two legs is oriented towardsthe illumination source. The document or document stack can be flippedover such that the bridging portion is facing upwards toward theillumination source. The document or document stack can be flipped byfirst moving one of the first document platform or the second documentplatform to be positioned under the document or document stack, thenmoving the other of the first document platform or second documentplatform over the document or document stack to sandwich the document ordocument stack between the two platforms. The two document platforms canbe rotated around a horizontal axis to flip over the document ordocument stack between the two document platforms. For example, the twoplatforms can simultaneously contact the document or document stack tosecurely position the document or document stack between the twoplatforms such that document or document stack remains positionedbetween the two platforms as the two platforms are rotated. Pressure maybe exerted upon the document or document stack by both of the platformsto ensure the document or document stack does not slip from between theplatforms when the platforms are rotated.

In some embodiments, one or more of the first document platform 260 orthe second document platform 270 can be a multi-axis component,including for example a two-axis or a three-axis component.

The first and/or second document platforms 260, 270 may be sized toaccommodate the document or document stack handled by the system 200. Insome embodiments, each of the first and second document platforms maycomprise a degree of roughness on a surface such that the surfaceconfigured to be in contact with the document or document stack can havedesired friction with the document or document stack. Providing adesired degree of friction on one or more surfaces of the first and/orsecond platforms may facilitate manipulation of the document or documentstack, and/or removal of one or more fasteners from the document ordocument stack. As described herein, it may be advantageous to maintainor substantially maintain the document or document stack in a desiredposition on one of the first or second platforms, such as during removalof one or more fasteners from the document of document stack.Maintaining or substantially maintaining the document or document stackin a desired position on a document platform can enable successfulremoval of the fasteners. It may be desired to securely position thedocument or document stack between the two platforms, such as whenrotating the two platforms to flip the document or document over. Thedesired friction can be achieved using various techniques. For example,friction can be provided on a surface of one or both of the platformsconfigured to contact the document or document stack by features formedthereon, including for example various recess patterns etched into thesurface. In some embodiments, the surface can have a coating to providethe desired friction. In some embodiments, a material of the platformcan be selected to provide the desired friction. In some embodiments,friction can be provided by placing additional material on the surface,such as strips of material configured to provide added friction. In someembodiments, a surface on one or both of the platforms configured tocontact the document or document stack can have a coefficient offriction of greater than about 0.5, about 0.6 or about 0.7. In someembodiments, the coefficient of friction can be about 0.5 to about 1,including about 0.6 to about 1, or about 0.7 to about 1.

In some embodiments, the document and fastener identification system 200can be configured to process a document or document stack having afastener attached thereto, such as from initiating illumination of thedocument or document stack to completing removal of the fastener, inless than about 15 seconds (sec), about 10 sec, about 5 sec, or about 3sec. In some embodiments, the process can be about 3 sec to about 15sec, including about 3 to about 10 sec, about 5 sec to about 15 sec orabout 10 sec to about 15 sec. For example, the processes can be anautomated process. The process may not include operator input after thedocument or document stack is positioned on the pedestal 250, at aninput of the system, or after illumination process of the document ordocument stack has initiated. In some embodiments, the system can beconfigured to process a document or document stack having an invertedfastener attached thereto, such as such as from initiating illuminationof the document or document stack to completing removal of the fastener,in less than about 20 seconds (sec), about 15 sec, about 10 sec, orabout 5 sec. In some embodiments, the process can be about 5 sec toabout 20 sec, including about 5 to about 15 sec. For example, theprocesses can be an automated process. The process may not includeoperator input after the document or document stack is positioned on thepedestal, at an input of the system, or after illumination process ofthe document or document stack has initiated.

Only one robot arm 240 is shown in FIG. 2. In some embodiments, use of asingle robot arm can reduce the footprint of the document and fasteneridentification system 200, reduce costs of operation and/or maintenance,and/or reduce sources of possible malfunction to thereby increase systemreliance. However, it will be understood that in some embodiments asystem for identifying a single document and/or fastener attached to adocument or document stack, and/or removal of the fastener, can includemore than one robot arm. In some embodiments, multiple robot arms can beincluded. Various tools, such as end effectors, can be coupled to therobot arms to manipulate the document and/or fastener, including forexample, to remove the fastener. In some embodiments, such a system caninclude one or more robot arms for manipulating the document or documentstack and one or more other robot arms to manipulate a fastener on thedocument or document stack.

FIGS. 3A through 3D are examples of unprocessed image captured by animaging device of document and fastener identification system, such asthe document and fastener identification system 200 of FIG. 2.Meanwhile, FIG. 3E is an example of a processed image generated by thedocument and fastener identification system, for example usinginformation of the unprocessed image of FIGS. 3A through 3D. Theplurality of unprocessed images of FIGS. 3A through 3E of the documentstack captured while the document stack is illuminated by illuminationmodules at different locations relative to the document stack can beused to generate the processed image shown in FIG. 3E. The unprocessedimages of FIG. 3D can be the image as captured by a digital camera. Theprocessed image of FIG. 3E can be an image generated by the document andidentification system, and shows features which are determined to bethree-dimensional features, such as the physical fasteners and/or edgesof one or more documents in the document stack, as identified by thedocument and fastener identification system.

One or more photometric stereovision algorithms can be used to processthe information from the unprocessed images, including for example usingvarious pattern recognition algorithms one skilled in the art wouldunderstand to be applicable. As described herein, a shadow profile ofthe document stack can be generated by comparing the unprocessed imagescaptured by the image capture device. For example, the comparison can bemade to determine whether any 3-D features are present in the image, as3-D features can cast shadows onto a surface of the document stackand/or a pedestal on which the document stack is positioned. Forexample, FIG. 3E can be generated by determining which features areactual three-dimensional features in the images. In some embodiments,information included in the processed image, such as the image of FIG.3E, can be further processed by the system to determine which of thethree-dimensional features are actual fasteners and/or boundaries ofdocuments, such that the fasteners and/or documents can be properlymanipulated. In some embodiments, one or more photometric stereovisionalgorithms can comprise algorithm configured to recognize shadowprofiles of various types of flexible material as described herein, suchas various types of paper. The one or more photometric stereovisionalgorithms may be configured to recognize various types of fasteners,including for example, various types of staples described herein. Theone or more photometric stereovision algorithms may be configured torecognize a position and/or an orientation of the fastener and/orflexible material. In some embodiments, a processed image comprises animage which shows the actual fasteners and/or document or documentstack, for example without showing other three-dimensional featureswhich are not actual fasteners and/or a boundary of the document and/ordocument stack.

In FIGS. 3A through 3D, the unprocessed images show a document 320 isshown positioned on a pedestal 350. The document 320 can include a firstboundary comprising outer edges 322 and a second boundary comprisinginner edges 324. The images show presence the appearance of a firststaple 302, a second staple 304, a third staple 306, a paper clip 308, abinder clip 310 on the document 320. The images also show a pair ofholes 312 on the document 310. For example, the pair of holes 312 can beholes made by a staple which has been removed. Each of the FIGS. 3Athrough 3D are images captured of the document when the document isilluminated by illumination modules at different locations relative tothe document.

FIG. 3E is a processed image generated by the document andidentification system showing the three-dimensional features captured bythe image capture device. In FIG. 3E, the second staple 304 is not shownin the processed image as the second staple is determined to not be athree-dimensional feature. For example, the second staple may be aphotocopied staple. FIG. 3E shows the first staple 302, third staple306, paper clip 308, binder clip 310, pair of holes 312 on the document320, and the inner edges 324 and outer edges 322 of the document, asthese features are determined to be three-dimensional features capturedin the images. FIG. 3E shows the various positions, and/or orientationsof the fasteners and edges of the document. The processed image may showfeatures from the pedestal 350 which are three-dimensional. As describedherein, the document and fastener identification system can beconfigured to further process the information shown in FIG. 3E.Information shown in FIG. 3E can be further processed, for example usingone or more pattern recognition algorithms, to determine which of thethree-dimensional features are part of a boundary of a document, and/orcorrespond to features of an actual fastener. The information can beprocessed to determine the type of the flexible material present, thetype of the fastener present, and/or various other characteristics ofthe document and/or fastener present (e.g., position, and/ororientation). For example, the system can be configured to determinethat the pair of holes is not part of an actual fastener on the documentand the features of the pedestal 350 are not features on the document.In some embodiments, a controller of a document and fasteneridentification system can be configured to, in generating instructionssignals for controlling one or more other components of the system forfastener removal, disregard information relating to three-dimensionalfeatures which do not correspond to a boundary of a document and/or donot correspond to a part of an actual fastener. For example, informationrelating to the pair of holes and the pedestal may be disregarded by thecontroller in determining control instructions for one or more othercomponents of the system.

In some embodiments, the document and fastener identification system canuse information generated by the document and fastener identificationsystem relating to the documents and fasteners to provide instructionsto one or robot arms for manipulating the documents and fasteners. Forexample, an appropriate tool, such as an end effector for coupling tothe robot arm, can be selected, and movement of the robot arm can becontrolled, to achieve desired movement of the papers and/or removal ofthe fasteners.

FIG. 4 is a schematic side view of an example of a document transfer endeffector 400. The document transfer end effector 400 can a have a body402 comprising a top portion 404 and a bottom portion 406. The topportion may comprise a robot coupling component 408 configured toreleasably couple the document transfer end effector to a robot arm,such as the robot arm 240 described with reference to FIG. 2. The robotcoupling component may comprise one or more of a number of mechanicalcoupling mechanisms (not shown) understood by a skilled artisan toenable releasable coupling between the document transfer end effectorand a robot arm. The robot coupling component can be configured to allowquick release from and/or engagement with the robot arm to enableincreased document processing speed. In some embodiments, the couplingcomponent comprises one or more electrical interfaces (not shown) fortransmission of electrical signals to and from the document transfer endeffector, such as to and from the robot arm for control of one or morecomponents of the document transfer end effector. The electricalinterfaces may be configured to transfer control signals from acontroller to the one or more components of the document transfer endeffector, such as to control the movement of the components. Thedocument transfer end effector can be stored at an end effector storageon one or more document and fastener identification systems describedherein when the document transfer end effector is not in use.

The lower portion 406 may comprise a suction applicator 410 configuredto lift at least a portion of a document or a document stack. Thesuction applicator may be configured to apply sufficient suction forceupon a portion of the document or document stack positioned on adocument pedestal, such as the pedestal 208 described with reference toFIG. 2, such that the document or document stack can be lifted andseparated from any other document(s) and/or document stack(s) on thepedestal. For example, the suction applicator may comprise a suctioncup.

The suction applicator 410 may extend from a lower surface 412 of thedocument transfer end effector body 402. The suction force exerted bythe suction applicator 410 may be selected to allow lifting of onedocument or document stack without lifting other document(s) or documentstack(s) on the pedestal to allow separation of the document or documentstack from the other document(s) or document stack(s). The suction forceapplied by the suction applicator can be configured to lift one documentand/or a plurality of documents. In some embodiments, informationcollected by a document and identification system regarding a documentor document stack can be used to determine the suction force applied,for example based on a number of documents in the document stack.

To apply the suction force upon the document or document stack, thedocument transfer end effector 400 can be brought proximate to thedocument or document stack such that the suction applicator 410 cancontact the document or document stack. A lower surface 414 of thesuction applicator can be brought into contact with the document ordocument stack such that desired suction force can be applied to thedocument or document stack. The suction applicator can be configured tocontact the document or document stack proximate to and/or at one ormore identified fasteners. In some embodiments, the suction applicatorcontacts at least a portion of the document or document stack to liftthe document or document stack, such as a portion of the document ordocument stack proximate to or at the identified fastener. For example,the document or document stack can be lifted by the document transferend effector due to suction force applied to a corner of the document ordocument stack. One or more fasteners, such as a staple, may be attachedto the document or document stack at the corner. Positioning of thesuction applicator at a desired location on the document or documentstack can be achieved using information collected regarding the documentor document stack. For example, a controller of a document and fasteneridentification system can be configured to use the collected informationcollected to control movement of the robot arm to which the documenttransfer end effector is coupled and/or the document transfer endeffector itself to make desired contact between the document transferend effector and the document or document stack.

In some embodiments, the suction applicator 410 can be extended and/orretracted relative to the body 402 to facilitate contact between thesuction applicator and the document or document stack. For example, thesuction applicator can be extended away from the body to bring thesuction applicator closer to the document or document stack, and thesuction applicator can be retracted back towards the body after thesuction applicator is no longer needed.

Referring to FIG. 4, the document transfer end effector 400 can comprisea document pincher 420 configured to facilitate lifting of the documentor document stack from a document pedestal and/or transfer of thedocument or document stack to a different location. As described herein,a document or document stack can be transferred onto a document platformfor removal of any fasteners attached thereto. The document pincher caninclude a rotatable coupling 422 for rotatably coupling the pincher tothe document transfer end effector body 402. The document pincher caninclude an arm 424 extending from the rotatable coupling. The arm canhave a first end 426 coupled to the rotatable coupling and a second end428 comprising a document gripper 430 extending therefrom. The rotatablecoupling can be rotated such that the arm is rotated relative to thebody to bring the document griper towards or away from the body. Thedocument gripper can be rotated towards the body to facilitate securingof the document or document stack against one or more portions of thedocument transfer end effector. For example, the suction applicator 410can apply a suction force upon the document or document stack such thatthe document or document stack can be lifted. The arm of the documentpincher can be rotated to bring the document gripper against towards thebody such that the document or document stack can be positioned betweenthe document gripper and one or more portions of the document transferend effector. The document transfer end effector may comprise a documentcontact platform 416 against which the document or document stack can bepositioned by the gripper. In some embodiments, the document transferend effector comprises the contact platform. In some embodiments, thedocument contact platform 416 may extend from the lower surface 412 ofthe body. In some embodiments, the document or document stack can bepinched between the document gripper and the document contact platform,such as between the document gripper and the lower surface 418 of thedocument contact platform. In some embodiments, the document or documentstack can be pinched between the document gripper and one or more otherportions of the document transfer end effector, including one or moreportions of the lower surface of the document transfer end effectorbody.

Use of the gripper 430 can advantageously provide secure lift of thedocument or document stack such that the document or document stack canbe reliably separated from any other document(s) or document stack(s)over which the document or document stack is placed. Securing thedocument or document stack using the gripper provides an added level ofreliability in the separation of the document or document stack, such ascompared to a system using the suction applicator 410 without thegripper.

The document gripper 430 may comprise a variety of configurations suitedto maintain the document or document stack against the body 402 of thedocument transfer end effector 400. In some embodiments, the documentgripper comprises a rod. For example, at least a portion of the documentgripper may comprise a rod. In some embodiments, the document grippercan comprise a different configuration, including for example a padconfiguration having a flat surface oriented towards the body 402. Thedocument gripper may have one or more configurations to maintain thedocument or document stack against the lower surface 412 of the bodysuch that desired transfer of the document or document stack can beachieved. In some embodiments, the document gripper can be configured tomaintain a hold on the document or document stack such that the documentpincher can be used to both pick up the document or document stack fromthe pedestal and transfer the document or document stack to a platform.The document or document stack can be positioned on the platform whenthe fastener removal is performed.

The document gripper 430 may comprise at least a portion which isoriented at an angle of less than about 180° from the second end 428 ofthe arm 424. In some embodiments, the document gripper comprises atleast a portion which is oriented at an angle of about 90° from thesecond end of the arm. The document gripper 430 may extend partiallyacross a dimension of the body 402, such as a width of the body. Thedocument gripper may extend across an entire or substantially entiredimension of the body. In some embodiments, the document grippercomprising a rod which extends perpendicularly or substantiallyperpendicularly from the second end of the arm. In some embodiments, therod may extend across an entire or substantially entire width of thebody. For example, the rod may be coupled to the body via two arms andtwo rotatable couplings 422, each arm and rotatable coupling on opposingportions of the body. In some embodiments, a rod extending across anentire or substantially entire width of the body may be coupled to thebody via one arm and one rotatable coupling. In some embodiments, therod may be perpendicular or substantially perpendicular to the arm andextend partially along a width of the body.

FIGS. 5A and 5B are schematic diagrams of an example of a fastenerremoval end effector 500. The fastener removal end effector may beconfigured to remove a staple from a document stack. The fastenerremoval end effector may be coupled to a robot arm, such as a robot arm240 described with reference to FIG. 2. FIG. 5A is front view of thefastener removal end effector and FIG. 5B is a side view of the fastenerremoval end effector. Referring FIG. 5A, the fastener removal endeffector can have a body 502 comprising a top portion 504 and a bottomportion 506. The top portion may comprise a robot coupling component 508configured to releasably couple the fastener removal end effector to arobot arm, such as the robot arm described with reference to FIG. 2. Therobot coupling component may comprise one or more of a number ofcoupling mechanisms understood by a skilled artisan to enable releasablecoupling between the fastener removal end effector and a robot arm. Forexample, the robot coupling component 508 can have one or more featuresof the robot coupling component 408 described with reference to FIG. 4.The robot coupling component can be configured to allow quick releasefrom and/or engagement with the robot arm to enable increased documentprocessing speed, such that the robot arm can switch between differentend effectors based on the task to be completed. In some embodiments,the coupling component comprises one or more electrical interfaces (notshown) for transmission of electrical signals to and from the fastenerremoval end effector, such as to and from the robot arm for control ofone or more components of the fastener removal end effector.

The fastener removal end effector 500 can include a first and a secondfoot structure 510, a fastener removal insert 530 and a fastener pincher540. The first and second foot structure, fastener removal insert andfastener pincher can extend from the lower portion 506 of the body 502.The fastener removal insert can be positioned below the fastenerpincher. The fastener removal insert and the fastener pincher can extendfrom a center or substantially a center portion of the body. The firstand second foot structure can be positioned on both sides of thefastener removal insert and the fastener pincher. In some embodiments,the first foot structure and the second foot structure can be used tocontact a document or document stack such that the document or documentstack is firmly held against a surface upon which the document ordocument stack is placed. The fastener removal insert can be insertedbetween a fastener attached to the document or document stack, such as astaple. The fastener pincher can be positioned against the fastenerremoval insert to firmly grip the fastener between the fastener pincherand the fastener removal insert. The fastener pincher and removal insertmay then be moved away from the document or document stack while thefirst and second foot structures hold the document or document stackagainst the surface upon which the document or document stack rests suchthat the fastener can be removed therefrom. For example, the fastenerpincher and removal insert may then be moved away from the document ordocument stack such that a staple on the document or document stack canbe pulled out.

Referring to FIG. 5A, the first and second foot structures 510 can eachcomprise a pair of elongate portions 512 which extends from the body502. The side view in FIG. 5B of the fastener removal end effector 500shows the pair of elongate portions for one of the first or second footstructure. The elongate portions can each have a first end 514 proximateto the body and a second opposing end 516 away from the body. The secondopposing ends of the foot structures can be coupled to respectivedocument contacting portions 518. The elongate portions can be extendedaway from the body to bring the document contacting portions intocontact with the document or document stack, and can be retractedtowards the body to release the document or document stack. For example,the elongate portions can be extended such that the document or documentstack can be firmly held in place between a bottom surface of thecontacting portions and the surface on which the document or documentstack is resting. In some embodiments, the first and second footstructures can be spring-loaded.

The document contacting portions 518 can have various configurations toprovide desired contact between the document contacting portions and thedocument or document stack such that the document and document stack canbe held firmly in place while the fastener is removed therefrom. In someembodiments, the contacting portions can comprise a document contactingsurface 520 oriented away from the body 502 and towards the document ordocument stack. As shown in FIG. 5B, in some embodiments, a contactingportion can be coupled to both elongate portions 512 of the footstructure 510. The contacting portion may comprise a rectangular orsubstantially rectangular configuration. For example, the documentcontacting surface can have a rectangular or substantially rectangularshape. In some embodiments, the contacting surface can have a length thesame or substantially the same as a corresponding dimension of the body,for example extending the entire or substantially entire correspondinglength of the body. In some embodiments, the document contacting surfacecan have one or more other shapes, such as a shape comprising a curvededge.

The fastener removal insert 530 can comprise a tab 532 having aconfiguration suited for inserting between the fastener and the documentor document stack. For example, the tab can be configured for insertionbetween a staple and the document or document stack. In someembodiments, the tab can comprise a thin flat protrusion. The fastenerremoval insert can be moved relative to the document or document stacksuch that at least a portion of the tab is inserted between the fastenerand the document or document stack. In some embodiments, the tab cancomprise at least a portion which has a thickness less than about 3millimeters (mm) in thickness, including less than about 2 mm, or about1 mm. In some embodiments, the tab can comprise a portion having thesame or substantially the same width as a width of the portion of thefastener under which the portion of the tab is inserted. In someembodiments, the tab can have a width greater than about 50% of acorresponding dimension of the fastener, including greater than about55%, about 60%, about 70%, about 80%, about 90% or about 95%. Forexample, the tab can have a width that is greater than about 95% thewidth of the portion of the fastener under which the tab is inserted.For example, the tab can be configured to be inserted under a portion ofa staple, and the portion of the tab configured to be insertedunderneath the staple can have a width greater than about 95% of theportion of the staple. In some embodiments, the tab can have a widthsuch that the tab is in contact with two ends of the staple extendinginto the document or document stack when the tab is inserted between thestaple and the document or document stack. In some embodiments, using atab comprising a portion with a width similar to that of a correspondingwidth of the fastener can facilitate desired removal of the fastener.For example, using a tab comprising a portion with a width similar tothat of a corresponding width of staple, such as the crown of thestaple, may facilitate removal of the staple without breaking thestaple. The staple may be removed without breaking off any portion ofthe staple which extends into the document or document stack.

The fastener pincher 540 may have a fastener gripper 542 configured tobe in contact with the fastener. For example, the fastener gripper canbe moved such that it contacts the fastener once the tab 532 of thefastener removal insert 530 is inserted between the fastener and thedocument or document insert. The fastener gripper can have a variety ofconfigurations suited for making firm contact with the fastener,including a planar or substantially planar surface configured to contacta corresponding surface of the fastener. In some embodiments, theportion of the fastener gripper in contact with the fastener can have adimension the same as or similar to a corresponding dimension of thefastener. In some embodiments, the portion of the fastener gripper incontact with the fastener can have a width greater than about 50% of acorresponding dimension of the fastener, including greater than about55%, about 60%, about 70%, about 80%, about 90% or about 95%. Forexample, the portion of the fastener gripper can have a width that isgreater than about 95% the width of the portion of the fastener withwhich the gripper is in contact. In some embodiments, using a grippercomprising a portion with a width similar to that of a correspondingwidth of the fastener can facilitate desired removal of the fastener.For example, using a gripper comprising a portion with a width similarto that of a corresponding width of staple, such as a width of the crownof the staple, may facilitate removal of the staple without breaking thestaple. The staple may be removed without breaking off any portion ofthe staple which extends into the document or document stack.

In some embodiments, a portion of the fastener, such as the crown of astaple, can be firmly sandwiched between the fastener removal insert 530and the fastener pincher 540. For example, the crown of a staple can besecurely squeezed by the fastener removal tab 532 and the fastenergripper 542, including a fastener removal tab and fastener gripperhaving a similar or same width as the crown, such that movement of thefastener removal tab and the fastener gripper away from the document ordocument stack held in place by the document contacting portions 518 canremove the staple without or substantially without breaking the staple.For example, the entire staple can be removed without leaving anyportion in the document or document stack. In some embodiments, firmlypinching the crown of the staple using the tab and the gripperfacilitates removal of both legs of the staple from the document ordocument stack at the same or similar speed, thereby reducing oreliminating breakage of the staple during the removal process. Achievingsuch reliable removal of the staple can provide a document processingsystem which can demonstrate reduced errors, lowered operating costs,and/or involve decreased operator intervention.

FIG. 6 is a schematic cross-sectional view of an example of a multi-taskend-effector 600. The multi-task end effector can be configured toperform both document transfer and fastener removal, such as thedocument transfer and fastener removal as described with reference toFIGS. 4 and 5. The multi-task end effector can be configured to includefunctionality of both the document transfer end effector 400 and thefastener removal end effector. In some embodiments, the multi-task endeffector can be configured to remove a staple. For example, one or morecomponents of the multi-task end effector for transferring a document ordocument stack can have one or more features of the document transferend effector of FIG. 4. One or more components of the multi-task endeffector for fastener removal can have one or more features of thefastener removal end effector of FIG. 5. A robot arm coupled to themulti-task end effector would not switch between different end effectorsto achieve both the document transfer and fastener removal, therebyreducing process time used to retrieve and/or return an end effector.

The multi-task end-effector can have a body 602, a suction applicator604, a document pincher 620, a document contacting platform 610, afastener removal insert 630, a fastener pincher 640, and a documentcontacting leg 650. The multi-task end effector can comprise a couplingcomponent (not shown) for coupling to a robot arm. The couplingcomponent comprising one or more features as described herein, includingfor example one or more mechanical and/or electrical features forcoupling to the robot arm. The document contacting platform can extendfrom the body. In some embodiments, the body comprises the documentcontacting platform. For example, a lower surface of the body comprisesthe lower surface 612 of the document contacting platform. The suctionapplicator can be moved toward a document or document stack, such asmoving in a direction away from the document contacting platform and/orthe body. The suction applicator can be moved such that a lower surface606 of the suction applicator can contact the document or document stackto apply suction force upon the document or document stack. In someembodiments, as described herein, the suction force can be applied toone or more portions of the document or document stack proximate to orat a fastener. Sufficient suction force can be applied to lift thedocument or document stack, for example to separate the document ordocument stack from any other documents and/or documents on a pedestalof a system. The suction applicator can be retracted toward the bodyand/or a lower surface of the document contacting platform after suctionforce is applied to the document or document stack, bringing thedocument or document stack towards the body and/or document contactingplatform.

The document pincher 620 can be subsequently actuated such that at leasta portion of the document pincher is positioned below the lifteddocument or document stack such that the document or document stack canbe securely positioned between the pincher and one or more othersurfaces of the multi-task end effector. For example, the documentpincher can comprise a first end 622 and a second end 624, the secondend being proximate to the suction applicator.

The document pincher can include a document gripper 626 coupled to thesecond end. The document gripper can be moved such that it is positionedunder the lifted document or document stack to firmly position thedocument or document stack between the gripper and one or more of thelower surface 606 of the suction applicator 604, the lower surface 612of the document contacting platform 610, and/or one or more othersurfaces of the end effector, including one or more other surfaces ofthe body of the end effector.

After the document is transferred to a desired position, the fastenerremoval components of the multi-task end effector 600 can be used toremove one or more fasteners on the document or document stack. Themulti-task end effector can include a fastener removal insert 630 forinsertion between a fastener and a top surface of the document ordocument stack. For example, the fastener removal insert may compriseone or more components and/or dimensions as described herein for removalof staples. The multi-task end effector can include a fastener pincher640 for positioning over and in contact with a fastener. For example,the fastener pincher may comprise one or more components and/ordimensions as described herein for removal of staples. The fastenerremoval insert and the fastener pincher may be configured to squeezetherebetween a fastener, such as a staple, such that movement of thefastener pincher and the fastener removal insert away from the documentor document stack removes the fastener from the document or documentstack. In some embodiments, the multi-task end effector comprises adocument contacting leg 650 configured to be in contact with thedocument or document stack while the fastener is removed to facilitateholding the document or document stack in place for the removal. Forexample, the document contacting leg can comprise a lower surface 562configured to be in contact with a top surface of the document ordocument stack during fastener removal. The document contacting leg canbe configured to apply pressure to the document or document stack suchthat the document or document stack is securely sandwiched between thedocument contacting leg and a top surface of the pedestal on which thedocument or document stack is placed.

In some embodiments, the lower surface 652 of the document contactingleg 650 can have a shape configured to provide desired force upon thedocument or document stack. For example, the lower surface can have a“U” shape or substantially a “U” shape. In some embodiments, thedocument contacting leg can have a lower portion comprising a “U” shapeor substantially a “U” shape. For example, the three sides of theU-shaped lower portion is placed proximate to the fastener to allow thefastener removal insert 630 and fastener pincher 640 to access thefastener within a space within the three sides, while the documentcontacting leg holds the document or document stack in place.

FIG. 7 is a schematic diagram of an example of an infrared fasteneridentification system 700. The infrared fastener identification systemcan use the infrared energy from a fastener to identify its positionand/or orientation on a document or document stack. The infraredfastener identification system can include a document pedestal 702 andan infrared camera 704. A document or document stack can be received onthe document pedestal. The infrared camera can be used to capture theinfrared energy of the document or document stack. The infrared energyimage can be processed by the system to generate thermal energyinformation of the document or document stack, including thermal energyof one or more fasteners attached to the document or document stack.

The infrared camera 704 can be any number of infrared cameras oneskilled in the art would understand as being suited for detection offasteners. Thermal energy radiated by a fastener may be different fromthat of the document(s). The infrared camera can be configured tocollect the thermal energy of a document or document stack beingprocessed, including thermal energy of any fasteners attached thereto.The infrared fastener identification system 700 can be configured to usethe difference in thermal energy between the flexible material of thedocument and the fastener to detect the presence of any fasteners and/oridentify the type of fasteners present. The system can be configured toaccurately quantify thermal energy of documents and/or fasteners thatare processed to facilitate identification of any number of differenttypes of fasteners that are attached to the documents being processed.

FIG. 8 is a schematic diagram of an example of a lift and scan fasteneridentification system 800. The lift and scan fastener identificationsystem can include a document pedestal 802, a lifter 804 and a scanner806. The document or document stack to be processed can be received onthe document pedestal. The lifter can be configured to lift the documentor document stack up from the document pedestal. The lifter may beconfigured to lift the document or document stack using varioustechniques, including using various mechanical techniques. In someembodiments, the lifter can comprise a plurality of suction applicatorsconfigured to apply suction force upon the document or document stack.In some embodiments, the lifter comprises four suction applicatorsconfigured to be brought into contact with four different portions of atop surface of the document or document stack positioned on thepedestal. The four suction applicators can be used to lift the documentor document stack. Subsequently, the scanner can be a used to scan forany obstruction in planes underneath the document, or a first documentof the document stack, to identify the presence of any fasteners and/orthe location of the fasteners that are present. In some embodiments, thescanner may comprise a laser scanner, such as a 2D laser scanner. Forexample, the 2D laser scanner may be oriented such that the scanner canscan for obstructions in planes underneath one or more portions of thedocument that have been lifted by the suction applicator.

A single document, with no fasteners thereon, would be lifted by thefour suction applicators, thereby leaving no obstruction underneath thedocument. A document stack comprising a fastener thereon can compriseone or more obstructions in the planes underneath the first document.For example, some or all of the first document of the stack may belifted by the suction applicators. Any portion of the first documentwhich may droop due to the weight of the fastener and/or being attachedto additional document(s) by the fastener can be detected by the scanner706. For example, the scanner can detect that obstructions may exist inthe planes underneath a plane of the first document lifted by thesuction applicator, such as due to one or more portions of the firstdocument drooping due to the weight of the fastener and/or otherdocument(s) of the stack attached thereto, and/or obstructions from oneor more portions of the other document(s) in the stack being lifted duethe attachment by the fastener. Such a technique can enable accuratedetermination of the location of a fastener, such as in which quadrantthe fastener is located, including fasteners which may be invisible toother types of analysis. In some embodiments, such a lift and scantechnique may advantageously facilitate identification of fasteners suchas adhesives which may not be visible from a top down surface analysisof the document or document stack. In some embodiments, the lift andscan technique can be used in combination with one or more other methodsdescribed herein. For example, the lift and scan technique can be usedto accurately determine a location of the fastener such that asubsequent more robust fastener identification analysis can be used tofocus on the identified location. In some embodiments, the lift and scantechnique can be combined with one or more stereovision processesdescribed herein. For example, a stereovision process may be used tofocus analysis on a particular portion of a document stack identified bythe lift and scan technique to determine the type and/or orientation offastener.

FIG. 9 is a schematic diagram of an inductive sensing fasteneridentification system 800. The inductive sensing fastener identificationsystem can include a document pedestal 902 and an inductive sensor 904.The inductive sensor may comprise an array of inductive sensors. Theinductive sensor can be configured to be brought into proximity of adocument or document stack positioned on the pedestal. The inductivesensor and/or the document pedestal 902 may be moved such that theinductive sensor can be passed over the entire document or documentstack to determine whether any fasteners are present. The inductivesensor 902 may be used to determine presence of any metallic fastenerson the document or document stack. In some embodiments, the inductivesensor can be used to provide an estimate of the location of thefasteners. For example, if a sensor in the array of inductive sensors istriggered to indicate presence of a fastener, the fastener can beapproximated to a particular distance from the triggered sensor. In someembodiments, the fastener or portion of the fastener can be less thanabout 4 centimeters (cm) from the senor, including less than about 3 cm,about 2 cm or about 1 cm from the triggered sensor.

In some embodiments, the inductive sensing technique as described hereincan be used in combination with one or more other methods describedherein. For example, the inductive sensing technique can be used toapproximate a location of the fastener such that a subsequent morerobust fastener identification analysis can be used to focus on theidentified location. In some embodiments, the inductive sensingtechnique can be combined with one or more stereovision processesdescribed herein. For example, a stereovision process may be used tofocus analysis on a particular portion of a document stack identified bythe inductive sensing technique to determine the type and/or orientationof fastener.

In some embodiments, the fastener identification systems 700, 800 and900 can include one or more other features as described herein,including for example one or more components configured to manipulatethe document or document stack. In some embodiments, the fasteneridentification systems 700, 800 and 900 can comprise one or moredocument platforms, and/or robot arms, such as robot arms configured tobe releasably coupled to one or more end effectors as described hereinfor the removal of fasteners.

One or more systems described herein may be used to fully automate aprocess for identification of single sheets of documents and/orfasteners. In some embodiments, one or more systems described herein canoperate within a larger automated process, such as part of an assemblyline in an automated warehouse used to digitize information stored onphysical documents. For example, one or more systems described hereincan form a robotic work cell or part of a robotic work cell. An assemblyline for digitizing information stored on physical documents cancomprise a plurality of robotic work cells. Such an assembly line can beinstalled to handle a large volume of documents. For example, anautomated facility may be able to process thousands of boxes ofdocuments a day. Automation of the document processing can improveefficiency, such as enabling an inventory queueing system for boxes ofdocuments to be processed, as well as a storage for boxes that have beendigitized but are awaiting quality assurance of the processed pages, forexample prior to shredding. Such an automated facility may be desiredfor a large archive in which the unscanned backlog of boxes mayrepresent years and years of scanning. Such a facility would enablerapid scanning of selected boxes on demand and returning the box to thearchive for permanent archive.

FIG. 10 is an example of a document and fastener identification process1000 for identifying a single sheet of document and/or one or morefasteners attached to a document or document stack. In some embodiments,the document and fastener identification process 1000 comprises aphotometric stereovision technique. In some embodiments, at least a partof the process 1000 can be implemented using one or more documentidentification systems described herein, such as the document andfastener identification system 200 described with reference to FIG. 2.In block 1002, the document or document stack can be positioned on apedestal of the document and fastener identification system. In block1004, an illumination source and image capturing device of the documentand identification system can be activated to capture multiple images ofthe document or document stack. The illumination source may comprise aplurality of illumination modules, each of the modules being configuredto be individually illuminated sequentially. For example, theillumination source may comprise four illumination modules, each of thefour illumination modules comprising a plurality of LEDs. The fourillumination modules may be illuminated in sequence such that one ormore images of the document stack can be captured by the imaging devicewhen each of the illumination modules are illuminated. The fourillumination modules can be located at different positions relative tothe document or document stack to illuminate the document or documentstack from different positions. Shadow profiles of any three-dimensionalfeatures on the document or document stack can be determined usingimages of the document or document stack illuminated from differentpositions.

In block 1006, images of the document stack are processed to identifyedges of a single sheet of document and/or the orientation and positionof any fastener present on the document or document stack. For example,shadow profiles obtained from the captured images of one or more edgesof a single sheet of document and/or fasteners on the document ordocument stack can be used to identify the edges of a single sheet ofdocument and/or the orientation and position of the fastener.

FIG. 11 is an example of a fastener removal process 1100 for removing afastener attached to a document or document stack. For example, a stapleattached to a document or document stack can be removed. The fastenerremoval process 1100 can be performed using one or more document andfastener identification systems described herein, including for example,the document and fastener identification system 200 described withreference to FIG. 2. In block 1102, the document or document stack canbe positioned on a pedestal of the document and fastener identificationsystem. In block 1104, an illumination source and image capturing deviceof the document and identification system can be activated to capturemultiple images of the document or document stack. In block 1106, theorientation and position of the fastener on the document or documentstack can be identified. For example, shadow profiles of one or morefasteners on the document or document stack obtained from the capturedimages can be used by the document and fastener identification system todetermine the orientation and position of the fastener.

In block 1108, a robot arm comprising a document transfer end effectorcan be used to lift the document or document stack from the pedestal.Lifting the document or document stack can separate the document ordocument stack from any other document(s) or document(s) on thepedestal. In some embodiments, one or more document transfer endeffectors as described herein may be used to pick up the document stack,including for example, the document transfer end effector 400 asdescribed with reference to FIG. 4. The document transfer end effectorcan be positioned over the fastener using the robot arm. A suctionapplicator of the document transfer end effector can be brought intocontact with the document or document stack and activated, such that asuction force can be exerted upon the document or document stack. Thesuction force can be used to maintain contact between the document ordocument stack and the suction applicator such that the document ordocument stack can be lifted from the pedestal. In some embodiments, thedocument transfer end effector comprises a document pincher configuredto further facilitate lifting of the document or document stack. Thedocument pincher can be activated such that a document gripper of thepincher can be brought into contact with the document or document stackto secure the document or document stack against a bottom surface of theend effector. For example, a document pincher can comprise a rotatablearm configured to bring the document gripper into contact with thedocument or document stack when the arm is rotated such that thedocument or document stack is pinned between the document gripper and alower surface of the end effector.

In block 1110, a document platform can be positioned beneath thedocument or document stack. Positioning the document platform beneaththe document or document stack can maintain separation of the documentor document stack from any document(s) or document stack(s) on thepedestal.

In block 1112, a robot arm comprising a fastener removal end effectorcan be used to remove the fastener from the document or document stack.One or more fastener removal end effectors described herein can be used,including the fastener removal end effector 500 as described withreference to FIG. 5. For example, a pair of document contacting portionsof the fastener removal end effector can be brought into contact with atop surface of the document or document stack. The document or documentstack may be held in position against the document platform using thedocument contacting portions. In some embodiments, a fastener removalinsert of the end effector can be inserted between a portion of thefastener and the top surface of the document or document stack. Forexample, a tab portion of fastener removal insert can be positionedbetween a crown portion of a staple and the top surface of the documentor document stack. A fastener gripper of the end effector cansubsequently be brought into contact with at least a portion of thefastener, such as at least a portion of the crown of a staple. Thefastener gripper can be pressed against the portion of the fastener,such as the portion of the crown. The fastener gripper and the tabportion of the end effector can then be moved away from the document ordocument stack such that the staple can be pulled out.

In some embodiments, the fastener gripper and/or the tab portion cancontact the entire or substantially the entire width of the crownportion of the staple, such that the staple can be reliably pulled outwithout breaking the staple.

FIG. 12 is another example of a fastener removal process 1200. Thefastener removal process 1100 can be performed using one or moredocument and fastener identification systems described herein, includingfor example, the document and fastener identification system 200described with reference to FIG. 2. The fastener removal process 1200can be used to remove an inverted fastener. For example, the process1200 can be used to remove an inverted staple attached to a document ordocument stack, such as a staple having its two bent legs facing theillumination source of a document and fastener identification systemrather than the crown portion extending between the two legs. In block1202, the document or document stack can be positioned on a pedestal ofthe document and fastener identification system. In block 1204, anillumination source and image capturing device of the document andidentification system can be activated to capture multiple images of thedocument or document stack. In block 1206, the fastener can beidentified as being inverted using the captured images. In block 1208,the document or document stack can be lifted from the pedestal. Liftingthe document or document stack can separate the document or documentstack from any other document(s) or document stack(s) on the pedestal.In block 1210, a first document platform can be positioned beneath thedocument or document stack. Positioning the first document platformbeneath the document or document stack can maintain separation betweenthe document or document stack from the other document(s) and/ordocument stack(s) on the pedestal. For example, the document or documentstack can be position on and in contact with the first documentplatform. In block 1212, a second document platform can be positionedover the document or document stack. The second platform can be incontact with the document or document stack such that the document ordocument stack is sandwiched between and in contact with the firstdocument platform and second document platform.

In block 1214, the first document platform and the second documentplatform can be rotated around a horizontal axis such that the documentor document stack therebetween is flipped over. Flipping the document ordocument stack over reorients the fasteners so that it is no longerinverted, thereby facilitating removal of the fastener. In block 1216, arobot arm coupled to a fastener removal end effector can be used toremove the fastener from the document or document stack. For example, afastener removal end effector, such as the fastener removal end effector500 described with reference to FIG. 5, can be used to remove thestaple.

While some embodiments of the present invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the invention. It should be understoodthat various alternatives to the embodiments of the invention describedherein may be employed in practicing the invention. Further, it isintended that methods and structures within the scope of the followingclaims and their equivalents be covered thereby.

1.-12. (canceled)
 13. A method for document processing, comprising: (a)providing a document transfer end effector, wherein the documenttransfer end effector comprises (i) a body, (ii) a suction applicatordisposed adjacent to a bottom portion of the body, and (iii) a documentpincher comprising an arm and a document gripper, the arm coupling thedocument gripper to the body; (b) contacting the suction applicator witha first surface of a document or document stack to subject the firstsurface to a suction force; (c) lifting the document or document stackin contact with the suction applicator; (d) contacting the documentgripper to a second surface of the document or document stack toposition the document or document stack between the document gripper andthe bottom portion of the bod, wherein the second surface is oppositethe first surface; and (e) moving the document transfer end effector totransfer the document or document stack from a first location to asecond location different from the first location.
 14. The method ofclaim 13, wherein (b) comprises extending or retracting the suctionapplicator relative to the body to facilitate contact between thesuction applicator and the first surface of the document or documentstack.
 15. The method of claim 13, wherein the suction applicatorcomprises a suction cup.
 16. The method of claim 13, wherein thedocument transfer end effector further comprises a robot couplingcomponent disposed adjacent to a top portion of the body, wherein therobot coupling component is releasably coupled to a robot arm, andwherein (e) comprises moving the robot arm to move the document transferend effector from the first location to the second location.
 17. Themethod of claim 13, further comprising, prior to (b), collectinginformation about the document or document stack to determine thesuction force to be applied.
 18. The method of claim 13, furthercomprising, prior to (b), collecting information about the document ordocument stack to identify a location of one or more fasteners attachedto the document or document stack, and, in (b), contacting the suctionapplicator at a portion of the document or document stack proximate toor at the location of the one or more fasteners.
 19. The method of claim13, wherein the arm is rotatably coupled to the body to permit thedocument gripper to rotate relative to the body to bring the documentgripper towards or away from the second surface.
 20. The method of claim13, wherein the document transfer end effector further comprises adocument contact platform disposed adjacent to the bottom portion of thebody, configured to interface with the document or document stack. 21.The method of claim 13, wherein the first location comprises a documentpedestal and the second location comprises a document platform.
 22. Themethod of claim 13, wherein the document gripper comprises at least aportion which is oriented at an angle of less than about 180° from anend of the arm coupled to the document gripper.
 23. A system fordocument processing, comprising: a document transfer end effector,wherein the document transfer end effector comprises (i) a body, (ii) asuction applicator disposed adjacent to a bottom portion of the body,and (iii) a document pincher comprising an arm and a document gripper,the arm coupling the document gripper to the body; and a controller unitin operable communication with the document transfer end effector,wherein the controller unit is configured to transfer a document ordocument stack from a first location to a second location by: contactingthe suction applicator with a first surface of the document or documentstack to subject the first surface to a suction force; lifting thedocument or document stack in contact with the suction applicator;contacting the document gripper to a second surface of the document ordocument stack to position the document or document stack between thedocument gripper and the bottom portion of the body, wherein the secondsurface is opposite the first surface; and moving the document transferend effector to transfer the document or document stack from the firstlocation to the second location.
 24. The system of claim 23, wherein thesuction applicator is extendable or retractable relative to the bottomsurface of the body to facilitate contact between the suction applicatorand the first surface of the document or document stack.
 25. The systemof claim 23, wherein the suction applicator comprises a suction cup. 26.The system of claim 23, wherein the document transfer end effectorfurther comprises a robot coupling component disposed adjacent to a topportion of the body, wherein the robot coupling component is releasablycoupled to a robot arm, and wherein the controller unit is in operablecommunication with the robot arm to move the document transfer endeffector from the first location to the second location.
 27. The systemof claim 23, wherein the controller unit is further configured tocollect information about the document or document stack to determinethe suction force to be applied.
 28. The system of claim 23, wherein thecontroller unit is further configured to collect information about thedocument or document stack to identify a location of one or morefasteners attached to the document or document stack, and, contact thesuction applicator at a portion of the document or document stackproximate to or at the location of the one or more fasteners.
 29. Thesystem of claim 23, wherein the arm is rotatably coupled to the body topermit the document gripper to rotate relative to the body to bring thedocument gripper towards or away from the second surface.
 30. The systemof claim 23, wherein the document transfer end effector furthercomprises a document contact platform disposed adjacent to the bottomportion of the body, configured to interface with the document ordocument stack.
 31. The system of claim 23, wherein the first locationcomprises a document pedestal and the second location comprises adocument platform.
 32. The system of claim 23, wherein the documentgripper comprises at least a portion which is oriented at an angle ofless than about 180° from an end of the arm coupled to the documentgripper.